A standard bi-encoder embeds an entire query into one fixed-length vector and an entire passage into another single fixed-length vector, then compares just those two vectors. ColBERT instead keeps a separate embedding for every token in the query and every token in the passage, and scores a query-passage pair with the MaxSim operation: for each query token, take its highest similarity to any token in the passage, then sum those per-token maximums across the whole query. What does this token-level 'late interaction' design let ColBERT capture that a single-vector bi-encoder cannot?
- It lets ColBERT skip computing passage representations in advance, since MaxSim can only be computed once the query is known, which removes the need for a pre-built index entirely
- It lets fine-grained matches on individual important terms surface in the score, because each query token can find its own best-matching passage token independently, instead of the whole query and the whole passage first being compressed into single vectors that can blur or lose the contribution of any one specific term
- It lets ColBERT skip the embedding model entirely and compare passages using exact string matching on the token text, since the highest-similarity token is always the token with the identical spelling
- It lets the passage side of the comparison be computed after the query arrives, rather than in advance, which reduces indexing time at the cost of slower per-query search
Why B? And why not the others?
Correct answer: B. It lets fine-grained matches on individual important terms surface in the score, because each query token can find its own best-matching passage token independently, instead of the whole query and the whole passage first being compressed into single vectors that can blur or lose the contribution of any one specific term
Compressing an entire query and an entire passage into a single vector each forces many words' worth of meaning to share one fixed-length representation, which can blur or dilute the contribution of any one specific term, especially a rare or unusually important one; ColBERT's late interaction keeps a separate embedding per token on both sides and lets each query token independently find its own best-matching passage token via MaxSim, so a strong match on one specific important term still shows up clearly in the summed score even if the rest of the query and passage differ, a fine-grained signal a single blended vector comparison cannot preserve. The claim that ColBERT skips computing passage representations in advance is backwards; ColBERT's entire efficiency advantage over a full cross-encoder comes from being able to pre-compute and store every passage's token embeddings before any query arrives, so only the query side needs to be embedded at search time. ColBERT does not use exact string matching at all; MaxSim compares learned token embeddings by similarity, and two different words can still score highly similar to each other if the model has learned they are semantically related, exactly the opposite of exact-spelling matching. The passage side is computed and stored ahead of time, not after the query arrives, which is precisely what makes ColBERT far cheaper at query time than a cross-encoder that must jointly process the query and passage together for every pair.
Source: Khattab & Zaharia, 'ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT' (2020), arXiv:2004.12832