A team wants the exact-term-matching efficiency of an inverted index (the same data structure BM25 relies on) but with better handling of vocabulary mismatch than raw keyword overlap provides, without giving up sparse indexing for a fully dense embedding index. SPLADE (Formal et al., 2021) produces a sparse vector for each query and document using a transformer, trained with explicit sparsity regularization. What does SPLADE's sparse vector actually represent, and why does it help with vocabulary mismatch?
- Each dimension corresponds to a document ID rather than a vocabulary term, so the vector directly lists which other documents are most similar to this one
- Each dimension corresponds to a term in the vocabulary, and the model assigns nonzero weight not only to terms that literally appear in the text but also to related terms it predicts are relevant (term expansion), so a document can still be matched on a query term it never literally contains, while the vector stays sparse enough to use the same efficient inverted-index infrastructure as BM25
- The vector has exactly one nonzero dimension representing the single most important word in the text, discarding every other term entirely
- Each dimension is a dense, uninterpretable float produced the same way a standard bi-encoder produces its embedding, and 'sparse' only describes how the vectors are stored on disk, not their content
Why B? And why not the others?
Correct answer: B. Each dimension corresponds to a term in the vocabulary, and the model assigns nonzero weight not only to terms that literally appear in the text but also to related terms it predicts are relevant (term expansion), so a document can still be matched on a query term it never literally contains, while the vector stays sparse enough to use the same efficient inverted-index infrastructure as BM25
SPLADE's sparse vector has one dimension per vocabulary term, and its explicit sparsity regularization together with a log-saturation effect on term weights pushes most dimensions to zero while letting a transformer assign nonzero weight to terms it predicts are relevant to the text's meaning, even terms that never literally appear in it; that learned term expansion is exactly what lets a document be matched on a query term it does not literally contain, addressing a vocabulary-mismatch failure that plain BM25, which only ever weights terms actually present in the text, cannot fix on its own. Because the representation stays genuinely sparse, it can still be served through the same efficient inverted-index infrastructure BM25 relies on, rather than requiring a dense approximate-nearest-neighbor index. The option describing dimensions as document IDs is wrong because SPLADE's dimensions are vocabulary terms, not other documents. The option describing a single nonzero dimension is wrong because SPLADE produces many nonzero term weights per vector, not just one. The option describing the vector as a dense, uninterpretable float array is wrong because SPLADE's whole design point is that its dimensions remain interpretable as specific vocabulary terms and the vector itself is sparse in content, not merely in storage format.
Source: Formal, Piwowarski & Clinchant, 'SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking' (2021), arXiv:2107.05720