A team's embedding model produces 1536-dimensional vectors, and storing and comparing vectors at full length is expensive at their corpus scale. They discover their embedding model was trained using Matryoshka Representation Learning (Kusupati et al., 2022), which lets them simply truncate each vector down to its first 256 dimensions and still get a useful representation, without retraining anything. What makes this truncation trick work, when truncating an ordinarily-trained embedding model's vector would badly damage its quality?
- The model was trained twice, once at the full dimension and once at the smaller dimension, and truncation simply switches which of the two independently trained vectors is used
- The later dimensions of the vector are trained to contain pure random noise on purpose, so removing them cannot remove any real information
- The model is trained so that information is organized coarse-to-fine across the dimensions, with each nested prefix of the vector, not just the full vector, optimized to be a usable representation on its own, so truncating to a shorter prefix still yields a meaningful embedding rather than an arbitrarily damaged one
- Truncation is only ever applied to the query vector at search time and never to the stored document vectors, so the two sides of every comparison are always at different lengths by design
Why C? And why not the others?
Correct answer: C. The model is trained so that information is organized coarse-to-fine across the dimensions, with each nested prefix of the vector, not just the full vector, optimized to be a usable representation on its own, so truncating to a shorter prefix still yields a meaningful embedding rather than an arbitrarily damaged one
Matryoshka Representation Learning trains a single model so that information is encoded coarse-to-fine across the vector's dimensions, explicitly optimizing many nested prefixes of the full vector, not just the complete vector, to each stand on their own as a usable representation; because of this training objective, simply keeping the first 256 of 1536 dimensions still yields a meaningful, independently useful embedding, whereas truncating a conventionally-trained model's vector would discard dimensions that were never individually optimized to carry a self-contained representation and would badly damage quality. The paper reports large storage and speed gains (up to roughly 14x smaller size and 14x faster large-scale retrieval at comparable accuracy) with no additional cost imposed at inference time. The option describing two independently trained models is wrong because Matryoshka Representation Learning produces one model whose single training run yields many usable prefix lengths, not two separate models. The option describing the later dimensions as deliberately pure noise is wrong because those dimensions still carry real, useful fine-grained information; they simply are not required for a coarser but still meaningful representation. The option restricting truncation to only the query side is wrong because the technique is meant to shrink stored vectors on both the query and document sides symmetrically, which is exactly what produces the storage and speed savings.
Source: Kusupati, Bhatt, Rege, Wallingford, Sinha, Ramanujan, Howard-Snyder, Chen, Kakade, Jain & Farhadi, 'Matryoshka Representation Learning' (2022), arXiv:2205.13147