A RAG pipeline's first stage retrieves the 50 candidate passages whose embeddings are closest to the query, using a bi-encoder that embeds the query and each passage independently and compares the two embeddings afterward. Before generation, a second-stage model re-scores those 50 candidates by feeding the query and each passage together into a single transformer that outputs one relevance score per pair. Why is this second-stage model typically applied only to a shortlist rather than to the entire document collection?
- It cannot process any text that has already been converted into an embedding vector, so it can only ever run before an embedding index exists
- It produces scores that are only meaningful when compared against a bi-encoder's scores, so by definition it must always run after the bi-encoder stage
- Scoring a query together with a passage in a single joint pass is far more computationally expensive per pair than comparing two independently pre-computed embeddings, so running it against every document in a large collection would be too slow; restricting it to a small shortlist keeps the added latency manageable while still improving ranking accuracy where it matters most
- It can only ever reproduce the same ranking the first-stage retrieval already produced, so applying it to the full collection would be redundant
Why C? And why not the others?
Correct answer: C. Scoring a query together with a passage in a single joint pass is far more computationally expensive per pair than comparing two independently pre-computed embeddings, so running it against every document in a large collection would be too slow; restricting it to a small shortlist keeps the added latency manageable while still improving ranking accuracy where it matters most
A cross-encoder feeds the query and a candidate passage through a transformer together, letting attention operate across both texts jointly, which produces a more accurate relevance judgment than comparing two embeddings computed in isolation, but that joint pass has to be run separately for every query-passage pair and cannot be pre-computed the way a document embedding can; running it against an entire large collection for every query would be prohibitively slow, so it is applied only to a small shortlist produced by a cheaper first-stage retriever, trading a modest amount of coverage for a large gain in per-pair accuracy exactly where it counts most. The option claiming it cannot process previously embedded text is wrong because a cross-encoder works directly on raw query-and-passage text pairs; nothing about the technique is restricted to running before an embedding index exists, it is a cost and design choice, not a technical limitation. The option describing a required dependency on bi-encoder scores is wrong because a cross-encoder produces a standalone relevance score with no need to be interpreted relative to any other model's output. The option claiming it would reproduce the same ranking is wrong because the entire value of reranking comes from a cross-encoder's joint attention frequently changing the order the first-stage retriever produced, which is exactly why it improves accuracy.
Source: Nogueira & Cho, 'Passage Re-ranking with BERT' (2019), arXiv:1901.04085