Data Sculptures Artwork by Sean Patrick Morris
All Tools

Embedding Drift Monitor

Quantify how much an embedding space changes when the underlying model is updated, swapped, or retrained. Given the same corpus embedded under two models, EDM runs a six-metric battery and classifies overall severity.

Embedding Drift Monitor on GitHub

  • Python
  • Embeddings
  • Drift
  • Monitoring
  • Machine Learning

Metric battery

All six metrics run by default. Each is independently classified as LOW, MODERATE, HIGH, or CRITICAL. The overall severity is the worst individual metric.

# Metric What it measures
1 Neighbourhood stability (Jaccard) What fraction of each point's k nearest neighbours are the same in both spaces? Score of 1.0 means local structure is perfectly preserved.
2 Neighbourhood rank correlation (Spearman) Among the common neighbours, do the distance rankings agree? Measures whether relative proximity is preserved, not just set membership.
3 Distance distribution shift (KS statistic) Do pairwise distances follow the same statistical distribution in both spaces? A high KS statistic means the global scale has shifted.
4 Global geometry (Mantel correlation) Is the pairwise distance matrix similar in both spaces? A high Mantel r means items that were far apart remain far apart.
5 Cluster membership stability (HDBSCAN + ARI) Do the natural clusters in one space correspond to those in the other? Adjusted Rand Index: 1.0 is a perfect match, 0.0 is random.
6 Hubness shift (Nk Spearman) Do the same items dominate as hub nodes (appearing frequently as a nearest neighbour)? A high correlation means hub structure is preserved.

Skip expensive metrics with --no-clusters, --no-geometry, --no-distance, or --no-hubs. HDBSCAN cluster analysis is O(n²) in memory; for corpora larger than roughly 23,000 items EDM will warn that it may require more than 4 GB RAM.

Severity interpretation

Classification

  • LOWMinimal drift. Safe to deploy.
  • MODERATENotable drift. Review before deploying.
  • HIGHSignificant drift. Full review required.
  • CRITICALSevere drift. Do not deploy without investigation.

Default thresholds (Jaccard, higher is better)

  • ≥ 0.80LOW
  • ≥ 0.50MODERATE
  • ≥ 0.20HIGH
  • < 0.20CRITICAL

Override thresholds per-metric via a .edm.toml config file.

Install

pip install embedding-drift-monitor

Requires Python 3.11 or newer and a C compiler for the HDBSCAN dependency. On Windows, install Microsoft C++ Build Tools first.

Usage

# Full drift analysis: all six metrics
edm compare reference.npy candidate.npy

# JSON output with per-point arrays (required for identify-regions)
edm compare reference.npy candidate.npy --format json --include-per-point -o results.json

# Markdown report with region breakdown by label
edm compare reference.npy candidate.npy --labels labels.txt --format markdown -o report.md

# Skip slow metrics on a large corpus
edm compare reference.npy candidate.npy --no-clusters --no-geometry -k 5,10

# Fast single-metric check: Jaccard stability at k=10 only
edm quick reference.npy candidate.npy

# Convert saved JSON results to a different format
edm report results.json --format markdown -o report.md

# Per-region drift breakdown (requires --include-per-point in the compare run)
edm identify-regions results.json -l labels.txt

Key options for edm compare

  • -kComma-separated k values for neighbourhood analysis (default: 5,10,25,50)
  • -fOutput format: text, json, or markdown (default: text)
  • -oWrite output to file (default: stdout)
  • -lLabels file: one label per line, corpus row order
  • --include-per-pointEmbed per-point arrays in JSON output (required for identify-regions)
  • --exclude-nanDrop rows containing NaN or Inf before analysis
  • --sample-sizeMax pairs sampled for distance and geometry metrics (default: 5,000)
  • --configTOML file with custom severity thresholds
  • --no-clustersSkip HDBSCAN cluster-stability (most expensive metric)
  • --no-geometrySkip Mantel global-geometry metric

Input formats

Both matrices must have the same number of rows (one row per corpus item). Column counts may differ: comparing embeddings from models with different output dimensions is valid.

  • .npyNumPy binary. Fastest. No pickle: allow_pickle=False.
  • .npzNumPy compressed archive with exactly one array.
  • .csv / .tsvNumeric data with optional header row. 10 MB cap.

File size cap: 50 MB per embedding file.

The embedding trilogy

EDM is the third tool in a sequence built around the same embedding space:

  • LLE V2What structures exist in this space? Exploration.
  • RQBWhen I reduce this space, how much structure is preserved? Evaluation.
  • EDMWhen the underlying model changes, how much structure shifts? Monitoring.