MongoDB Research releases LEAF distillation framework: 23M-parameter embedding models keep up to 97% of teacher performance and run retrieval without a GPU
Synopsis
MongoDB Research has released LEAF, a knowledge distillation framework, together with two 23M-parameter text embedding models, mdbr-leaf-ir and mdbr-leaf-mt. State-of-the-art embedding models have grown to hundreds of millions or billions of parameters, typically need GPU servers, and are not compatible with one another, so swapping models means recomputing every document embedding. LEAF aligns only the teacher and student output embeddings; the distilled models retain up to 97% of their teacher's performance while being 5–15× smaller and 7–24× faster, and can run offline on CPU-only servers, IoT and mobile devices.
Interpretation
The student models distilled by LEAF are compatible with their larger teachers, so the text embeddings the two produce are interoperable. Information retrieval systems typically use the same large model for both document indexing and query encoding, and existing embedding models are not compatible with each other, even across sizes within one family, so swapping models requires recomputing every document embedding from scratch. The compatibility comes from a training objective whose sole goal is making the student's output embeddings approximate the teacher's, which supports asymmetric deployments where documents are embedded with a larger model and queries are encoded by a local, fast LEAF model.
LEAF constrains only the L2 approximation error between the teacher's and student's final output embeddings, without aligning intermediate layers. Early distillation methods tried to align as much of the teacher's and student's internals as possible, including attention matrices and the hidden states of all layers and attention heads; later work focused on the last layer of the Transformer backbone only. Training needs no labeled data and avoids the judgments and hard negatives that contrastive training requires; it favors small batches, and both distilled models were trained on a single A100 GPU.
The two 23M-parameter models each rank first on public leaderboards within their parameter tier. mdbr-leaf-ir was distilled from snowflake-arctic-embed-m-v1.5, and mdbr-leaf-mt from mxbai-embed-large-v1. mdbr-leaf-ir ranks first on the public BEIR and RTEB leaderboards for models with at most 100M parameters; mdbr-leaf-mt ranks first on the public MTEB v2 (Eng) leaderboard for models with at most 30M parameters.
Perspective
The demonstrated scope is text embedding models: the two 23M-parameter models are compared on the public BEIR, RTEB and MTEB v2 (Eng) leaderboards within stated parameter caps, throughput was measured on a 2vCPU instance at 120 queries/sec and 20 docs/sec, and the full-precision versions have an 87MB memory footprint. The models are released on Hugging Face under Apache 2.0 as is, without explicit or implied support, and at the time of writing are not used in any MongoDB commercial product or service.
The publisher describes the models as best-in-class for their size, a judgment based on public leaderboard rankings within parameter caps rather than cross-size comparison. Throughput and memory figures come from measurements on a specific instance, and the per-variant gains of the available quantizations are not itemized in the text.