RAG & Embeddings

Why A? And why not the others?

Correct answer: A. The knowledge base can be kept current by re-indexing the changed documents alone, without retraining or replacing the model's weights, so newly added or edited information becomes available to the chatbot as soon as it is indexed

Retrieval-augmented generation keeps the model's parameters untouched and instead updates a separate, continuously refreshed index of documents; because generation pulls whatever the retriever finds at query time, a policy change becomes usable the moment the changed document is re-indexed, with no retraining cycle in between, unlike fine-tuning, which requires collecting new training examples and running a training job every time the underlying documents change. The option claiming fine-tuning cannot teach a model any new facts overstates the case -- fine-tuning can shift a model's learned associations, it is simply slow, costly, and impractical to repeat every week, which is exactly why the comparison favors retrieval instead. The option describing an unlimited context window is wrong because retrieval augmentation still inserts retrieved text into a prompt that is bounded by the model's context window; that limit is precisely why only a small number of top-ranked passages, not the whole document set, are retrieved and inserted. The option promising guaranteed accuracy is wrong because retrieval only supplies the model with candidate passages; nothing about the retrieval step verifies the truth of those passages or forces the generated answer to stay faithful to them.

Source: Lewis et al., 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' (2020), arXiv:2005.11401

Why B? And why not the others?

Correct answer: B. It prevents a sentence or idea that spans a chunk boundary from being split apart so that neither resulting chunk contains it in full, which could leave either chunk incoherent or missing context on its own

Splitting a manual into non-overlapping chunks risks cutting a sentence, a step in a procedure, or a cause-and-effect explanation exactly at the boundary, leaving each side of that boundary incomplete on its own; giving adjacent chunks a modest overlap means the text spanning that seam appears in full inside at least one chunk, so retrieval is less likely to surface a fragment that is missing the context needed to make sense of it. The option about reducing the number of chunks is backwards: adding overlap increases the total number of chunks and the embedding cost, since the overlapping text is embedded more than once, rather than reducing it. The option promising guaranteed retrieval is wrong because appearing in an extra chunk does not guarantee a match; whether a chunk is retrieved still depends on how closely its embedding matches the query, not simply on how many times its text appears in the index. The option about processing shorter chunks confuses overlap with chunk size itself; the maximum length an embedding model can accept is a separate setting from how much adjacent chunks overlap.

Source: Pinecone, 'Chunking Strategies for LLM Applications,' https://www.pinecone.io/learn/chunking-strategies/; LangChain, 'Splitting recursively' text splitter documentation, https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter

Why C? And why not the others?

Correct answer: C. Nearest-neighbor search, which embeds the query with the same embedding model and retrieves the passages whose vectors are closest to the query's vector

Because the embedding model places passages with related meaning near each other in vector space regardless of shared vocabulary, the natural way to exploit that property is to embed the incoming query with the same (or a compatible) model and then search for the document vectors nearest to it, retrieving passages that are semantically close even when they use entirely different wording than the query. The option describing exact string matching is wrong because it depends on literal character overlap between query and document, which is precisely the limitation that embedding-based retrieval is designed to overcome. The option describing regular-expression matching against predefined templates is wrong because it can only recognize a fixed set of patterns chosen in advance and cannot generalize to novel phrasing the way a learned vector space can. The option describing sorting by publication date is wrong because it ignores the content and the embedding vectors entirely, so it would surface recent passages regardless of whether they have any semantic relationship to the query at all.

Source: Reimers & Gurevych, 'Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks' (2019), arXiv:1908.10084

Why D? And why not the others?

Correct answer: D. It measures the angle between two vectors rather than their length, so two embeddings pointing in the same direction score as highly similar even if one vector happens to have a larger magnitude than the other

Cosine similarity divides the dot product of two vectors by the product of their lengths, which cancels out each vector's magnitude and leaves only the cosine of the angle between them; this makes it well suited to comparing embeddings whose length can vary for reasons unrelated to meaning, such as differences in passage length, since two vectors pointing in nearly the same direction score as similar regardless of how long either one happens to be. The option claiming cosine similarity is always faster is wrong because computing it still requires essentially the same multiply-and-add arithmetic as a dot product, plus extra work to normalize by each vector's length; any speed difference in practice comes from indexing and hardware optimizations, not from the metric itself. The option describing a binary-vector conversion is wrong because cosine similarity operates directly on the original floating-point vectors rather than converting them into a binary representation. The option about requiring matching dimensionality is wrong because that requirement applies equally to the dot product and Euclidean distance, so it does not distinguish cosine similarity from the alternatives at all.

Source: Manning, Raghavan & Schutze, 'Introduction to Information Retrieval' (2008), Section 6.3, Vector Space Scoring, https://nlp.stanford.edu/IR-book/

Why A? And why not the others?

Correct answer: A. HNSW is an approximate nearest-neighbor algorithm: it searches a multi-layer navigable graph structure to reach a good answer quickly, accepting a small chance of missing the true nearest neighbor in exchange for search times far faster than comparing the query against every stored vector

HNSW builds a multi-layer graph in which higher layers provide long-range shortcuts and lower layers refine the search locally, letting a query traverse from a coarse starting point down to a close neighborhood in far fewer comparisons than checking every stored vector; this graph-based shortcut is what makes HNSW fast at large scale, and it is also exactly why the algorithm is described as approximate -- the greedy graph traversal can settle on a good-enough neighborhood without ever comparing against every vector, so it occasionally misses the single closest one that an exhaustive search would have found. The option about deleting near-duplicates is wrong because HNSW does not remove vectors it judges similar to others; every inserted vector remains a node in the graph. The option about indexing only a random sample is wrong because HNSW builds its graph over all inserted vectors, not a subset chosen in advance. The option about rounding coordinates describes vector quantization, a separate, optional compression technique that is not what defines HNSW's fundamental speed-versus-recall trade-off.

Source: Malkov & Yashunin, 'Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs' (2016), arXiv:1603.09320

Why B? And why not the others?

Correct answer: B. Add a sparse, keyword-based retrieval method such as BM25 alongside the dense embedding search, and combine both rankings into a single hybrid result set, since exact-term matching methods are specifically strong at the rare-token, exact-identifier queries where dense embeddings tend to struggle

Dense embedding models are trained to capture semantic similarity and tend to smooth over exact tokens such as rare identifiers, treating them as similar to other short alphanumeric strings rather than requiring an exact character match; a sparse, term-based method like BM25 scores documents on the exact terms they contain, so it is specifically good at surfacing a document because it contains the literal string 'ERR-4471,' and combining that ranking with the dense ranking in a hybrid setup covers the weakness that either method has on its own. The option about retraining on a larger general corpus is wrong because a bigger corpus does not change the fundamental tendency of a dense embedding to generalize over exact tokens; it would not reliably fix retrieval of a specific rare identifier. The option about increasing embedding dimensionality is wrong because more dimensions increase representational capacity in general but do not add a mechanism for exact lexical matching. The option about shrinking chunk size is wrong because isolating the code into its own chunk does not change how the embedding model represents it, so a dense search would still return approximate neighbors rather than an exact match.

Source: Robertson & Zaragoza, 'The Probabilistic Relevance Framework: BM25 and Beyond' (2009); Karpukhin et al., 'Dense Passage Retrieval for Open-Domain Question Answering' (2020), arXiv:2004.04906

Why C? And why not the others?

Correct answer: C. Scoring a query together with a passage in a single joint pass is far more computationally expensive per pair than comparing two independently pre-computed embeddings, so running it against every document in a large collection would be too slow; restricting it to a small shortlist keeps the added latency manageable while still improving ranking accuracy where it matters most

A cross-encoder feeds the query and a candidate passage through a transformer together, letting attention operate across both texts jointly, which produces a more accurate relevance judgment than comparing two embeddings computed in isolation, but that joint pass has to be run separately for every query-passage pair and cannot be pre-computed the way a document embedding can; running it against an entire large collection for every query would be prohibitively slow, so it is applied only to a small shortlist produced by a cheaper first-stage retriever, trading a modest amount of coverage for a large gain in per-pair accuracy exactly where it counts most. The option claiming it cannot process previously embedded text is wrong because a cross-encoder works directly on raw query-and-passage text pairs; nothing about the technique is restricted to running before an embedding index exists, it is a cost and design choice, not a technical limitation. The option describing a required dependency on bi-encoder scores is wrong because a cross-encoder produces a standalone relevance score with no need to be interpreted relative to any other model's output. The option claiming it would reproduce the same ranking is wrong because the entire value of reranking comes from a cross-encoder's joint attention frequently changing the order the first-stage retriever produced, which is exactly why it improves accuracy.

Source: Nogueira & Cho, 'Passage Re-ranking with BERT' (2019), arXiv:1901.04085

Why D? And why not the others?

Correct answer: D. Performance on tasks that require using information from within a long context follows a U-shaped curve, staying strongest for information near the very beginning or the very end of the context and degrading for information placed in the middle, so the most relevant of the 8 passages should be placed near the start or end of the prompt rather than buried in the middle

Studies of how language models use long input contexts found a consistent U-shaped performance curve on tasks that require locating and using a specific piece of information: accuracy is highest when the needed information sits near the beginning or the end of the context and drops noticeably when that information is placed in the middle, a pattern attributed to a mix of primacy and recency effects rather than uniform attention across all positions; the practical implication for a RAG prompt built from several retrieved passages is to place the passages judged most relevant near the start or end rather than in the middle of the stack. The option describing a hard length ceiling of only a few thousand tokens conflates this positional effect with a completely different constraint, a fixed context-length limit, when the scenario already assumes all 8 passages fit inside the prompt. The option claiming position has no effect is the opposite of what was documented. The option claiming only the very last passage matters overstates the finding, since the documented curve favors both the beginning and the end, not the end alone.

Source: Liu et al., 'Lost in the Middle: How Language Models Use Long Contexts' (2023), arXiv:2307.03172

Why A? And why not the others?

Correct answer: A. Recall@20 = 8/10 = 0.8, because 8 of the 10 relevant passages were retrieved; precision@20 = 8/20 = 0.4, because only 8 of the 20 returned passages were relevant

Recall@k is defined as the number of truly relevant items found within the top k results divided by the total number of truly relevant items that exist for that query, which here is 8 divided by 10, or 0.8; precision@k is defined as the number of truly relevant items found within the top k results divided by k itself, which here is 8 divided by 20, or 0.4. The option that swaps the two formulas -- dividing the relevant-and-retrieved count by 20 to get recall and by 10 to get precision -- is the classic error of confusing these two definitions and applying each one's arithmetic to the wrong metric. The option pooling everything into a denominator of 18 is wrong because neither metric's standard definition combines missed relevant passages and irrelevant retrieved passages into a single shared denominator; each metric has its own distinct denominator. The option claiming recall cannot be computed is wrong because recall's denominator is the total number of truly relevant passages for the query, a quantity already given as 10 in the scenario, not the unrelated and much larger size of the entire corpus.

Source: Manning, Raghavan & Schutze, 'Introduction to Information Retrieval' (2008), Chapter 8, Evaluation in Information Retrieval, https://nlp.stanford.edu/IR-book/

Why B? And why not the others?

Correct answer: B. The retrieval component is doing its job, since the needed information was present in what was retrieved, but the generation step is not staying grounded in that retrieved context and is producing claims the context does not actually support

Context recall and faithfulness are deliberately separate measurements precisely so that retrieval quality and generation quality can be diagnosed independently: a high context recall score means the necessary supporting information was present among the retrieved passages, so the retrieval half of the pipeline did what it needed to do, while a low faithfulness score means the generated answer nonetheless contains claims that are not actually supported by that same retrieved context, pointing to a generation-time grounding problem such as the model drifting from or embellishing beyond what the passages state. The option blaming missing retrieved information directly contradicts the premise that context recall is already high, meaning the needed information was in fact present in what was retrieved. The option recommending a newer embedding model misdiagnoses the fix, since embeddings affect what gets retrieved, a stage the scenario already says is working, not whether the generated text stays grounded in that retrieved material. The option claiming the two metrics always move together denies the entire reason both are reported separately; if they were redundant, reporting one score high and the other low would be impossible in the first place.

Source: Es et al., 'Ragas: Automated Evaluation of Retrieval Augmented Generation' (2023), arXiv:2309.15217

Why C? And why not the others?

Correct answer: C. It prompts a language model to generate a hypothetical answer to the query first, then embeds that generated hypothetical answer, rather than the original query, and uses that embedding to search the index, on the theory that a plausible answer is likely to use vocabulary closer to the real documents than the short original query does

HyDE addresses the vocabulary mismatch by having a language model first write a hypothetical answer to the query, essentially imagining what a document that answers the question might say, and then embedding that generated passage instead of the original short query; because a fabricated answer written in the style of the domain is likely to use phrasing closer to the actual documents, such as 'declined by issuing bank' rather than the user's own words, the resulting embedding lands closer to the relevant real documents in vector space than the original query's embedding would have. The option describing thesaurus-based synonym expansion is wrong because it is a much older, purely lexical technique that appends related words to the query text itself rather than generating and embedding a hypothetical document with a language model. The option describing per-query retraining of the embedding model is wrong because retraining a model before every individual query is computationally infeasible, and HyDE deliberately leaves the embedding model untouched. The option describing multilingual translation and per-language search is wrong because that addresses a mismatch between different languages, not the same-language vocabulary mismatch described here, and it is not what HyDE does at all.

Source: Gao, Ma, Lin & Callan, 'Precise Zero-Shot Dense Retrieval without Relevance Labels' (2022), arXiv:2212.10496

Why D? And why not the others?

Correct answer: D. Because the similarity search collects only a fixed-size pool of top-k candidates before any filtering happens, a filter that matches only a sparse subset of the index causes most of that fixed pool to be discarded, leaving too few results; pre-filtering avoids this by restricting the candidate set to metadata-matching vectors before or during the similarity search itself, so the search only ever ranks documents that could pass the filter in the first place

The shortfall comes from the order of operations: post-filtering first fixes the candidate pool at a small, constant size (the top-k by similarity) and only afterward removes candidates that fail the metadata filter, so when the matching subset is a sparse fraction of the whole index, most of that fixed-size pool gets thrown away, leaving too few results even though many matching documents exist elsewhere; pre-filtering fixes this by narrowing the set of vectors the similarity search is allowed to consider, either before the similarity search runs or woven directly into the search's own traversal, so every candidate the search finds already satisfies the filter. The option proposing an exact brute-force search is wrong because switching from approximate to exact search does not change the fact that only a fixed-size top-k pool is retrieved before filtering is applied; the same shortfall would recur. The option proposing embedding metadata directly into the semantic vector is wrong because mixing a categorical field into the same continuous space used for meaning would distort the semantic similarity the embedding is meant to capture, rather than fixing how filtering interacts with candidate pool size. The option declaring the situation unfixable is wrong because a real alternative, pre-filtering, exists, and pushing the burden onto applications to only ever write broad filters is both inaccurate and impractical for real use cases that genuinely need narrow ones.

Source: Pinecone, 'The Missing WHERE Clause in Vector Search,' https://www.pinecone.io/learn/vector-search-filtering/