RAG & Embeddings

Correct answer: .

Retrieval-augmented generation keeps the model's parameters untouched and instead updates a separate, continuously refreshed index of documents; because generation pulls whatever the retriever finds at query time, a policy change becomes usable the moment the changed document is re-indexed, with no retraining cycle in between, unlike fine-tuning, which requires collecting new training examples and running a training job every time the underlying documents change. The option claiming fine-tuning cannot teach a model any new facts overstates the case -- fine-tuning can shift a model's learned associations, it is simply slow, costly, and impractical to repeat every week, which is exactly why the comparison favors retrieval instead. The option describing an unlimited context window is wrong because retrieval augmentation still inserts retrieved text into a prompt that is bounded by the model's context window; that limit is precisely why only a small number of top-ranked passages, not the whole document set, are retrieved and inserted. The option promising guaranteed accuracy is wrong because retrieval only supplies the model with candidate passages; nothing about the retrieval step verifies the truth of those passages or forces the generated answer to stay faithful to them.

Source: Lewis et al., 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' (2020), arXiv:2005.11401

Correct answer: .

Splitting a manual into non-overlapping chunks risks cutting a sentence, a step in a procedure, or a cause-and-effect explanation exactly at the boundary, leaving each side of that boundary incomplete on its own; giving adjacent chunks a modest overlap means the text spanning that seam appears in full inside at least one chunk, so retrieval is less likely to surface a fragment that is missing the context needed to make sense of it. The option about reducing the number of chunks is backwards: adding overlap increases the total number of chunks and the embedding cost, since the overlapping text is embedded more than once, rather than reducing it. The option promising guaranteed retrieval is wrong because appearing in an extra chunk does not guarantee a match; whether a chunk is retrieved still depends on how closely its embedding matches the query, not simply on how many times its text appears in the index. The option about processing shorter chunks confuses overlap with chunk size itself; the maximum length an embedding model can accept is a separate setting from how much adjacent chunks overlap.

Source: Pinecone, 'Chunking Strategies for LLM Applications,' https://www.pinecone.io/learn/chunking-strategies/; LangChain, 'Splitting recursively' text splitter documentation, https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter

Correct answer: .

Because the embedding model places passages with related meaning near each other in vector space regardless of shared vocabulary, the natural way to exploit that property is to embed the incoming query with the same (or a compatible) model and then search for the document vectors nearest to it, retrieving passages that are semantically close even when they use entirely different wording than the query. The option describing exact string matching is wrong because it depends on literal character overlap between query and document, which is precisely the limitation that embedding-based retrieval is designed to overcome. The option describing regular-expression matching against predefined templates is wrong because it can only recognize a fixed set of patterns chosen in advance and cannot generalize to novel phrasing the way a learned vector space can. The option describing sorting by publication date is wrong because it ignores the content and the embedding vectors entirely, so it would surface recent passages regardless of whether they have any semantic relationship to the query at all.

Source: Reimers & Gurevych, 'Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks' (2019), arXiv:1908.10084

Correct answer: .

Cosine similarity divides the dot product of two vectors by the product of their lengths, which cancels out each vector's magnitude and leaves only the cosine of the angle between them; this makes it well suited to comparing embeddings whose length can vary for reasons unrelated to meaning, such as differences in passage length, since two vectors pointing in nearly the same direction score as similar regardless of how long either one happens to be. The option claiming cosine similarity is always faster is wrong because computing it still requires essentially the same multiply-and-add arithmetic as a dot product, plus extra work to normalize by each vector's length; any speed difference in practice comes from indexing and hardware optimizations, not from the metric itself. The option describing a binary-vector conversion is wrong because cosine similarity operates directly on the original floating-point vectors rather than converting them into a binary representation. The option about requiring matching dimensionality is wrong because that requirement applies equally to the dot product and Euclidean distance, so it does not distinguish cosine similarity from the alternatives at all.

Source: Manning, Raghavan & Schutze, 'Introduction to Information Retrieval' (2008), Section 6.3, Vector Space Scoring, https://nlp.stanford.edu/IR-book/

Correct answer: .

HNSW builds a multi-layer graph in which higher layers provide long-range shortcuts and lower layers refine the search locally, letting a query traverse from a coarse starting point down to a close neighborhood in far fewer comparisons than checking every stored vector; this graph-based shortcut is what makes HNSW fast at large scale, and it is also exactly why the algorithm is described as approximate -- the greedy graph traversal can settle on a good-enough neighborhood without ever comparing against every vector, so it occasionally misses the single closest one that an exhaustive search would have found. The option about deleting near-duplicates is wrong because HNSW does not remove vectors it judges similar to others; every inserted vector remains a node in the graph. The option about indexing only a random sample is wrong because HNSW builds its graph over all inserted vectors, not a subset chosen in advance. The option about rounding coordinates describes vector quantization, a separate, optional compression technique that is not what defines HNSW's fundamental speed-versus-recall trade-off.

Source: Malkov & Yashunin, 'Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs' (2016), arXiv:1603.09320

Correct answer: .

Dense embedding models are trained to capture semantic similarity and tend to smooth over exact tokens such as rare identifiers, treating them as similar to other short alphanumeric strings rather than requiring an exact character match; a sparse, term-based method like BM25 scores documents on the exact terms they contain, so it is specifically good at surfacing a document because it contains the literal string 'ERR-4471,' and combining that ranking with the dense ranking in a hybrid setup covers the weakness that either method has on its own. The option about retraining on a larger general corpus is wrong because a bigger corpus does not change the fundamental tendency of a dense embedding to generalize over exact tokens; it would not reliably fix retrieval of a specific rare identifier. The option about increasing embedding dimensionality is wrong because more dimensions increase representational capacity in general but do not add a mechanism for exact lexical matching. The option about shrinking chunk size is wrong because isolating the code into its own chunk does not change how the embedding model represents it, so a dense search would still return approximate neighbors rather than an exact match.

Source: Robertson & Zaragoza, 'The Probabilistic Relevance Framework: BM25 and Beyond' (2009); Karpukhin et al., 'Dense Passage Retrieval for Open-Domain Question Answering' (2020), arXiv:2004.04906

Correct answer: .

A cross-encoder feeds the query and a candidate passage through a transformer together, letting attention operate across both texts jointly, which produces a more accurate relevance judgment than comparing two embeddings computed in isolation, but that joint pass has to be run separately for every query-passage pair and cannot be pre-computed the way a document embedding can; running it against an entire large collection for every query would be prohibitively slow, so it is applied only to a small shortlist produced by a cheaper first-stage retriever, trading a modest amount of coverage for a large gain in per-pair accuracy exactly where it counts most. The option claiming it cannot process previously embedded text is wrong because a cross-encoder works directly on raw query-and-passage text pairs; nothing about the technique is restricted to running before an embedding index exists, it is a cost and design choice, not a technical limitation. The option describing a required dependency on bi-encoder scores is wrong because a cross-encoder produces a standalone relevance score with no need to be interpreted relative to any other model's output. The option claiming it would reproduce the same ranking is wrong because the entire value of reranking comes from a cross-encoder's joint attention frequently changing the order the first-stage retriever produced, which is exactly why it improves accuracy.

Source: Nogueira & Cho, 'Passage Re-ranking with BERT' (2019), arXiv:1901.04085

Correct answer: .

Studies of how language models use long input contexts found a consistent U-shaped performance curve on tasks that require locating and using a specific piece of information: accuracy is highest when the needed information sits near the beginning or the end of the context and drops noticeably when that information is placed in the middle, a pattern attributed to a mix of primacy and recency effects rather than uniform attention across all positions; the practical implication for a RAG prompt built from several retrieved passages is to place the passages judged most relevant near the start or end rather than in the middle of the stack. The option describing a hard length ceiling of only a few thousand tokens conflates this positional effect with a completely different constraint, a fixed context-length limit, when the scenario already assumes all 8 passages fit inside the prompt. The option claiming position has no effect is the opposite of what was documented. The option claiming only the very last passage matters overstates the finding, since the documented curve favors both the beginning and the end, not the end alone.

Source: Liu et al., 'Lost in the Middle: How Language Models Use Long Contexts' (2023), arXiv:2307.03172

Correct answer: .

Recall@k is defined as the number of truly relevant items found within the top k results divided by the total number of truly relevant items that exist for that query, which here is 8 divided by 10, or 0.8; precision@k is defined as the number of truly relevant items found within the top k results divided by k itself, which here is 8 divided by 20, or 0.4. The option that swaps the two formulas -- dividing the relevant-and-retrieved count by 20 to get recall and by 10 to get precision -- is the classic error of confusing these two definitions and applying each one's arithmetic to the wrong metric. The option pooling everything into a denominator of 18 is wrong because neither metric's standard definition combines missed relevant passages and irrelevant retrieved passages into a single shared denominator; each metric has its own distinct denominator. The option claiming recall cannot be computed is wrong because recall's denominator is the total number of truly relevant passages for the query, a quantity already given as 10 in the scenario, not the unrelated and much larger size of the entire corpus.

Source: Manning, Raghavan & Schutze, 'Introduction to Information Retrieval' (2008), Chapter 8, Evaluation in Information Retrieval, https://nlp.stanford.edu/IR-book/

Correct answer: .

Context recall and faithfulness are deliberately separate measurements precisely so that retrieval quality and generation quality can be diagnosed independently: a high context recall score means the necessary supporting information was present among the retrieved passages, so the retrieval half of the pipeline did what it needed to do, while a low faithfulness score means the generated answer nonetheless contains claims that are not actually supported by that same retrieved context, pointing to a generation-time grounding problem such as the model drifting from or embellishing beyond what the passages state. The option blaming missing retrieved information directly contradicts the premise that context recall is already high, meaning the needed information was in fact present in what was retrieved. The option recommending a newer embedding model misdiagnoses the fix, since embeddings affect what gets retrieved, a stage the scenario already says is working, not whether the generated text stays grounded in that retrieved material. The option claiming the two metrics always move together denies the entire reason both are reported separately; if they were redundant, reporting one score high and the other low would be impossible in the first place.

Source: Es et al., 'Ragas: Automated Evaluation of Retrieval Augmented Generation' (2023), arXiv:2309.15217

Correct answer: .

HyDE addresses the vocabulary mismatch by having a language model first write a hypothetical answer to the query, essentially imagining what a document that answers the question might say, and then embedding that generated passage instead of the original short query; because a fabricated answer written in the style of the domain is likely to use phrasing closer to the actual documents, such as 'declined by issuing bank' rather than the user's own words, the resulting embedding lands closer to the relevant real documents in vector space than the original query's embedding would have. The option describing thesaurus-based synonym expansion is wrong because it is a much older, purely lexical technique that appends related words to the query text itself rather than generating and embedding a hypothetical document with a language model. The option describing per-query retraining of the embedding model is wrong because retraining a model before every individual query is computationally infeasible, and HyDE deliberately leaves the embedding model untouched. The option describing multilingual translation and per-language search is wrong because that addresses a mismatch between different languages, not the same-language vocabulary mismatch described here, and it is not what HyDE does at all.

Source: Gao, Ma, Lin & Callan, 'Precise Zero-Shot Dense Retrieval without Relevance Labels' (2022), arXiv:2212.10496

Correct answer: .

The shortfall comes from the order of operations: post-filtering first fixes the candidate pool at a small, constant size (the top-k by similarity) and only afterward removes candidates that fail the metadata filter, so when the matching subset is a sparse fraction of the whole index, most of that fixed-size pool gets thrown away, leaving too few results even though many matching documents exist elsewhere; pre-filtering fixes this by narrowing the set of vectors the similarity search is allowed to consider, either before the similarity search runs or woven directly into the search's own traversal, so every candidate the search finds already satisfies the filter. The option proposing an exact brute-force search is wrong because switching from approximate to exact search does not change the fact that only a fixed-size top-k pool is retrieved before filtering is applied; the same shortfall would recur. The option proposing embedding metadata directly into the semantic vector is wrong because mixing a categorical field into the same continuous space used for meaning would distort the semantic similarity the embedding is meant to capture, rather than fixing how filtering interacts with candidate pool size. The option declaring the situation unfixable is wrong because a real alternative, pre-filtering, exists, and pushing the burden onto applications to only ever write broad filters is both inaccurate and impractical for real use cases that genuinely need narrow ones.

Source: Pinecone, 'The Missing WHERE Clause in Vector Search,' https://www.pinecone.io/learn/vector-search-filtering/

Correct answer: .

Semantic chunking places a chunk boundary wherever the similarity between consecutive sentences drops, which is exactly the point where the text moves from one topic to another; because the boundary tracks the content rather than an arbitrary character count, each resulting chunk stays about one topic, so its embedding represents that topic rather than an average of several unrelated ones, and a query about just one of those topics is more likely to land close to the chunk in vector space instead of being diluted by unrelated material stuffed in alongside it. Cross-encoder reranking is a separate technique that re-scores already-retrieved candidates by feeding the query and passage through a transformer together; it has nothing to do with how chunk boundaries are chosen before indexing. Query expansion changes the text of the query or chunk by adding related terms, which is a lexical-augmentation idea unrelated to deciding where to cut a long document into chunks. Vector quantization compresses embeddings to save memory and speed up comparisons, and it is a lossy compression step that if anything can slightly reduce retrieval accuracy in exchange for speed, not a technique for choosing chunk boundaries at all.

Source: Pinecone, 'Chunking Strategies for LLM Applications,' https://www.pinecone.io/learn/chunking-strategies/; LangChain, semantic chunking documentation (percentile/standard-deviation breakpoint thresholds), https://python.langchain.com/docs/how_to/semantic-chunker/

Correct answer: .

MMR selects results one at a time, scoring each remaining candidate as a trade-off between how relevant it is to the query and how similar it is to results already chosen; the first pick is whichever candidate is most relevant, but every later pick is penalized for resembling an already-selected result, so once one passage restating a fact has been chosen, near-identical restatements of that same fact score poorly against the diversity penalty while passages covering different information score relatively better and rise into the result set instead. Deleting corpus passages within a fixed distance threshold ahead of time is not how MMR works; MMR operates on the ranked candidate list at query time rather than editing the underlying corpus, and a fixed global threshold would not adapt to which passages a particular query happened to retrieve. Retraining the embedding model to push similar passages apart is also not what MMR does; MMR leaves the embeddings and the corpus untouched and only changes which of the already-computed nearest neighbors get selected into the final result list. Splitting the corpus in half and interleaving results is an arbitrary partitioning scheme that has no connection to MMR's actual relevance-versus-redundancy trade-off and would not reliably avoid near-duplicate results at all.

Source: Carbonell & Goldstein, 'The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries' (1998), https://aclanthology.org/X98-1025.pdf

Correct answer: .

The question above actually asks for two distinct things -- what changed about pricing between two versions, and which of those changes mattered more to one customer segment -- and a single embedding of the whole sentence produces one vector that blends both asks together, which tends to retrieve passages that partially match one part or the other rather than passages that fully match either; query decomposition fixes this by splitting the compound question into narrower sub-questions first, retrieving separately for each one so that each search targets a single, coherent piece of information, and only combining the separately retrieved evidence at generation time. Simply increasing the number of retrieved passages does not fix the underlying problem, since the single blended query embedding is still what determines which passages are considered close matches in the first place; adding more of the same imprecisely matched candidates does not guarantee the missing, more precisely relevant ones appear. Switching to a higher-dimensional embedding model does not solve this either, since more dimensions increase how much information a vector can represent in general but do not give a single embedding of a multi-part question the ability to separately match multiple distinct pieces of information at once. Applying a stricter similarity cutoff on the original single search only removes the weakest matches from an already poorly targeted result set; it cannot introduce the more precisely relevant passages that a differently phrased, narrower query would have found.

Source: NVIDIA, 'Query Decomposition for NVIDIA RAG Blueprint,' https://docs.nvidia.com/rag/latest/query_decomposition.html; 'Question Decomposition for Retrieval-Augmented Generation' (2025), arXiv:2507.00355

Correct answer: .

Compressing an entire query and an entire passage into a single vector each forces many words' worth of meaning to share one fixed-length representation, which can blur or dilute the contribution of any one specific term, especially a rare or unusually important one; ColBERT's late interaction keeps a separate embedding per token on both sides and lets each query token independently find its own best-matching passage token via MaxSim, so a strong match on one specific important term still shows up clearly in the summed score even if the rest of the query and passage differ, a fine-grained signal a single blended vector comparison cannot preserve. The claim that ColBERT skips computing passage representations in advance is backwards; ColBERT's entire efficiency advantage over a full cross-encoder comes from being able to pre-compute and store every passage's token embeddings before any query arrives, so only the query side needs to be embedded at search time. ColBERT does not use exact string matching at all; MaxSim compares learned token embeddings by similarity, and two different words can still score highly similar to each other if the model has learned they are semantically related, exactly the opposite of exact-spelling matching. The passage side is computed and stored ahead of time, not after the query arrives, which is precisely what makes ColBERT far cheaper at query time than a cross-encoder that must jointly process the query and passage together for every pair.

Source: Khattab & Zaharia, 'ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT' (2020), arXiv:2004.12832

Correct answer: .

Self-RAG's reflection tokens are produced by the model itself as part of ordinary generation and serve two purposes that always-retrieve pipelines lack: a retrieval-decision signal that lets the model judge, per query, whether retrieving is even necessary versus answering directly, and critique signals that assess whether a given retrieved passage is actually relevant and whether the model's own generated statements are properly supported by it; together these let the system skip retrieval on queries like simple arithmetic where it adds nothing, and flag or discount passages and claims that don't hold up, rather than blindly retrieving for every query and blindly trusting whatever came back. The reflection tokens have nothing to do with cross-lingual retrieval or translation; Self-RAG's mechanism is about deciding whether and how to use retrieval, not about which language is searched. They also are not a summarization mechanism; nothing in Self-RAG's design compresses retrieved passages into shorter text before generation. And the reflection tokens do not retrain the retriever at query time; Self-RAG's critic and reflection-token behavior are learned once during training on a fixed dataset, not updated online from each new incoming query.

Source: Asai, Wu, Wang, Sil & Hajishirzi, 'Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection' (2023), arXiv:2310.11511

Correct answer: .

An embedding model's vector space is a byproduct of its own particular training process, so the geometric relationships that make similarity scores meaningful, such as which directions correspond to which shades of meaning, are specific to that one model; a vector produced by a different model, even one with an identical number of dimensions, was never placed into that same learned geometry, so measuring cosine similarity or a dot product between the two amounts to comparing coordinates from two unrelated coordinate systems that happen to share a dimension count, not two points meaningfully positioned relative to each other. This is why switching embedding models requires re-embedding the entire existing corpus with the new model rather than mixing old and new vectors in one index. The claim about precision mismatches always causing a runtime error is wrong; numeric precision differences do not by themselves make a similarity computation fail, and this is not the actual reason the comparison is meaningless. The claim that vector databases reject multiple models per collection at the software level is also wrong; most vector databases will happily store and compare vectors of matching dimensionality regardless of which model produced them, which is exactly why this mistake is possible to make silently rather than being blocked outright. And re-encoding the query once per document is not a real requirement of any embedding-based retrieval system; the query is embedded once and compared against many pre-computed document vectors, so this option describes a nonexistent step rather than the actual cause of the mismatch.

Source: OpenAI, embedding models documentation on model-specific vector spaces, https://developers.openai.com/api/docs/guides/embeddings; OpenAI Help Center, 'Embeddings FAQ,' https://help.openai.com/en/articles/6824809-embeddings-faq

Correct answer: .

GraphRAG targets exactly this gap between narrow fact-lookup and broad, corpus-wide sensemaking questions by building a knowledge graph of entities and relationships extracted from the corpus with an LLM, then partitioning that graph into communities of closely related entities and pre-generating a summary for each community; a broad question about overall themes can then be answered using these pre-built, higher-level community summaries, which already synthesize information spread across many source passages, rather than hoping the answer happens to sit inside whichever handful of passages a similarity search returns. Simply retrieving more raw passages does not solve the underlying problem, since the answer to a genuinely corpus-wide question is not contained in any single passage or a larger pile of individually retrieved passages to begin with, no matter how many are pulled in. Swapping in a larger embedding model improves how well individual passages are represented but does not create the cross-document synthesis a broad thematic question needs, since that synthesis does not exist inside any one passage's embedding. Fine-tuning the language model on the whole collection is a fundamentally different approach from GraphRAG, which deliberately keeps retrieval-based grounding rather than baking the corpus into the model's weights, and fine-tuning does not build the graph structure or community summaries that GraphRAG's method is centered on.

Source: Edge et al., 'From Local to Global: A Graph RAG Approach to Query-Focused Summarization' (Microsoft Research, 2024), arXiv:2404.16130

Correct answer: .

Pasting an entire document collection into the prompt for every single query means the model has to process that huge number of tokens over again on every call, and comparisons measuring this directly found the long-context approach costing many times more and taking many times longer per query than retrieving and sending only a handful of relevant passages, while answering about as accurately on the same workload; at any meaningful query volume, that cost-and-latency gap compounds because it is paid again on every single request, which is why RAG can remain the better engineering choice even once a model's context window is technically large enough to fit everything. The claim that a larger context window makes a model ignore retrieved information is not an accurate description of what was being compared or observed; the actual finding was about cost and speed, not about the model refusing to use provided context. The claim that context windows above roughly 100,000 tokens don't actually work is false; current models do support windows at that scale and well beyond, which is exactly what makes the whole-corpus-in-one-prompt approach possible in the first place, just expensive. And citations are not exclusive to RAG pipelines; a long-context prompt can also be instructed to cite which part of the pasted text supports its answer, so the ability to cite sources does not by itself distinguish the two approaches.

Source: Towards Data Science, 'Kimi K3's 1M Token Context Window vs. RAG: Cost, Latency and Answer Quality,' https://towardsdatascience.com/kimi-k3s-1m-token-context-window-vs-rag-cost-latency-and-answer-quality/; Redis, 'RAG vs Large Context Window: Real Trade-offs for AI Apps,' https://redis.io/blog/rag-vs-large-context-window-ai-apps/

Correct answer: .

Faithfulness and answer relevancy are deliberately measured as separate properties: faithfulness checks whether the claims in the generated answer are actually backed by the retrieved context, while answer relevancy checks whether the generated answer directly addresses the specific question that was asked, and these two things can diverge, exactly as in this scenario, where every statement made is well supported by the retrieved passages (high faithfulness) yet the answer wanders into adjacent territory instead of directly answering the question posed (low answer relevancy); this combination points at a generation-time problem with staying targeted to the actual question, not a grounding problem. The claim that the two scores contradict each other misunderstands what each one measures; being well supported by evidence and being on-topic for the specific question asked are independent properties, and an answer can easily have one without the other, as this scenario demonstrates directly. Blaming missing information in the retrieved passages misattributes what low answer relevancy actually measures; it evaluates the generated answer's focus against the question, not whether the retrieved context contained the needed facts, and nothing here indicates the passages were insufficient. And there is no rule making answer relevancy meaningful only when faithfulness is low; both metrics are reported and interpreted independently precisely so that a case like this one, high on one axis and low on the other, can be diagnosed rather than discarded.

Source: Es, James, Espinosa Anke & Schockaert, 'RAGAS: Automated Evaluation of Retrieval Augmented Generation' (2023), arXiv:2309.15217

Correct answer: .

Splitting a long document into chunks strips each chunk of its surrounding context, so a chunk that says something like 'the company's revenue grew 3% over the previous quarter' can be ambiguous in isolation about which company or which quarter is meant, and an embedding or keyword index built from that bare fragment ends up representing that ambiguity rather than the specific fact the source document actually intended; Anthropic's technique addresses this by having a model generate a short, chunk-specific blurb, such as naming the company and filing period, and prepending it to the chunk before embedding it for dense search and before adding it to the keyword index, so both retrieval methods see the disambiguated version rather than the bare fragment. The claim about a hard minimum embedding-model input length is not the motivation here; embedding models can generally process short chunks without needing to be padded up to some minimum size, and Anthropic's technique targets ambiguity in meaning, not an input-length constraint. The generated context does not replace the original chunk text; it is prepended alongside the chunk's own content specifically so the chunk's original information remains present together with the added disambiguating context. And the generated context is not merely a human-facing quality-assurance note that gets stripped out before indexing; it is deliberately kept in what gets embedded and keyword-indexed, since that is precisely the mechanism by which it improves retrieval.

Source: Anthropic, 'Introducing Contextual Retrieval,' https://www.anthropic.com/engineering/contextual-retrieval

Correct answer: .

BM25 scores a query term's contribution using an inverse-document-frequency weight, so a term that appears in only a few documents across the whole collection counts for more than a term that appears almost everywhere, which corrects for common, uninformative terms otherwise inflating scores just as much as distinctive ones. It also saturates the raw term-frequency contribution using a length-normalization parameter (commonly denoted b) that compares a document's length to the collection's average length, so a document cannot rack up an artificially high score purely by being long and repeating terms more often. The option about image counts and byte size is wrong because BM25 operates purely on term statistics within text, with no notion of embedded media. The option describing a dense-embedding cosine similarity is wrong because that describes a semantic retrieval method entirely separate from BM25, which stays lexical and never computes embeddings. The option restricting scoring to the title field is wrong because BM25 scores terms wherever they appear in the indexed text field, not only within a title.

Source: Robertson & Zaragoza, 'The Probabilistic Relevance Framework: BM25 and Beyond' (2009); mechanics corroborated via the Okapi BM25 reference entry (Wikipedia, cross-checked against the original paper's term-frequency saturation and IDF formulation)

Correct answer: .

RRF sidesteps the incompatible-scales problem entirely by ignoring raw scores and working only with rank position: for each document, it computes 1/(rank + k) within each list the document appears in (with k conventionally a small constant such as 60), then sums that value across all the lists that document shows up in, and sorts documents by the resulting total. Because this depends only on where a document placed in each list, it works identically whether the underlying scores are unbounded BM25 scores or bounded cosine similarities, with no normalization step required. The option describing discarding the lower-scoring list is wrong because RRF fuses information from every list rather than picking a single winner, and 'average score' is exactly the kind of raw-score comparison RRF is designed to avoid needing. The option describing retraining a single embedding model is wrong because RRF is a fusion algorithm applied after both searches already ran, not a model-training step. The option describing re-running a cross-encoder over the union of both lists is wrong because that describes cross-encoder reranking, a separate technique with its own computational cost, not RRF's rank-based fusion.

Source: Cormack, Clarke & Buettcher, 'Reciprocal Rank Fusion outperforms Condorcet and Individual Rank Learning Methods' (SIGIR 2009); mechanics corroborated via Microsoft Learn, Azure AI Search 'Hybrid search scoring (RRF)' documentation

Correct answer: .

This 'chunk expansion' (small-to-big) approach deliberately decouples the unit used for search from the unit delivered to the LLM: the small, precisely-scoped chunk stays the thing that gets embedded and matched against the query, preserving the similarity search's precision, but once that small chunk is identified as relevant, the surrounding text around it is pulled in and expanded up to a paragraph, page, or whole document before being handed to the LLM, restoring the context the bare fragment was missing. The option describing embedding every chunk at two sizes and picking whichever scores higher is wrong because it still forces a single tradeoff between search precision and context richness on the search side, rather than separating the two concerns as chunk expansion does. The option describing simply retrieving more small chunks is wrong because piling on more disconnected small fragments does not restore the specific surrounding context of any one retrieved chunk. The option describing fine-tuning the embedding model to pack more context into a small chunk's own vector is wrong because a fixed-length vector has limited capacity, and this approach does not address the fact that the chunk's actual text, not just its vector, is what the LLM needs more of.

Source: Pinecone, 'Chunking Strategies for LLM Applications,' https://www.pinecone.io/learn/chunking-strategies/

Correct answer: .

Product quantization replaces each subvector with the ID of its nearest centroid from a small codebook, and that centroid is only an approximation of the original subvector's actual values, so any similarity or distance computed from the compressed representation is itself an approximation of the true full-precision distance; documented benchmarks on this kind of index show recall dropping well below a full-precision flat index's near-perfect recall. A team accepts this because the compression can shrink an index's memory footprint by roughly an order of magnitude or more (letting a corpus that would not otherwise fit in memory fit at all) while combining an IVF partitioning stage with the quantized vectors also makes search dramatically faster than scanning everything exhaustively, since a query only has to compare against vectors in a handful of nearby partitions rather than the full corpus. This is a different mechanism from an HNSW index's own speed-versus-recall tradeoff, which comes from an approximate graph traversal that can skip a true nearest neighbor, not from lossily compressing the vectors' values themselves. The option claiming no tradeoff exists is wrong because quantization is lossy by construction. The option claiming only insertion ability is lost is wrong because the accuracy cost falls on search recall, not on whether new vectors can be added. The option claiming only metadata filtering is lost is wrong because metadata filtering is a separate concern from how the vectors themselves are compressed and searched.

Source: Pinecone (Faiss series), 'Product Quantization,' https://www.pinecone.io/learn/series/faiss/product-quantization/

Correct answer: .

RAPTOR builds a tree with differing levels of summarization from the bottom up by recursively embedding, clustering, and summarizing chunks, so that at query time the system can draw on higher, more abstractive levels of the tree that already synthesize many chunks' worth of content, instead of being limited to whatever a single flat chunk or a small handful of them happen to say; the paper reports a large absolute-accuracy gain on a complex-reasoning benchmark when this retrieval is combined with a strong generation model, specifically because broad questions benefit from these pre-built higher-level summaries. The option describing fine-tuning the language model on the corpus is wrong because RAPTOR keeps a retrieval-based architecture rather than baking corpus knowledge into model weights. The option describing extracting entities and relationships into a knowledge graph with community summaries describes a different technique (an entity-and-relationship graph approach), not RAPTOR's recursive clustering-and-summarization tree, which builds its hierarchy directly from chunk text and embeddings rather than from extracted entities. The option describing simply retrieving more raw chunks is wrong because it adds no new structure above the chunks and does nothing to synthesize information spread across many of them.

Source: Sarthi et al., 'RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval' (2024), arXiv:2401.18059

Correct answer: .

CRAG inserts a lightweight retrieval evaluator that assesses the overall quality of the retrieved documents for a given query and returns a confidence degree, and depending on that confidence, the pipeline can trigger a corrective knowledge-retrieval action, such as extending to a large-scale web search, rather than proceeding to generate an answer straight from documents the evaluator has already flagged as weak. This is a distinct mechanism from Self-RAG, where the model itself generates reflection tokens as part of its own output to judge, during generation, whether retrieval was needed or whether the passages it received were useful; CRAG instead uses a separate, dedicated evaluator that runs before generation begins and reacts with an external corrective action rather than the model reflecting on its own output. The option describing the model generating its own reflection tokens is therefore wrong, since that describes Self-RAG rather than CRAG's external evaluator. The option describing removing retrieval entirely for broad questions is wrong because CRAG's evaluator reacts to retrieval quality, not to whether a question is broad or narrow. The option describing re-embedding the whole corpus with a larger model on the fly is wrong because CRAG's corrective action is to seek additional or alternative sources such as web search, not to rebuild the existing index.

Source: Yan, Gu, Zhu & Ling, 'Corrective Retrieval Augmented Generation' (2024), arXiv:2401.15884

Correct answer: .

FLARE iteratively predicts the upcoming sentence to anticipate what content it will need, and if that anticipated sentence contains low-confidence tokens, it treats the anticipated content as a query, retrieves relevant documents, and regenerates the sentence using that newly retrieved context; this check repeats across the course of generation rather than retrieving only once based on the original input, which is what lets it adapt to a long answer whose later sentences need different sources than its first sentence did. The option describing retrieving after every single token regardless of confidence is wrong because FLARE's trigger is specifically low-confidence tokens in the anticipated upcoming sentence, not an unconditional per-token retrieval. The option describing a single check-and-replace pass after generation finishes is wrong because FLARE's retrieval decisions happen throughout generation, sentence by sentence, not as one final verification step. The option describing a human reviewer manually flagging sentences is wrong because FLARE's retrieval trigger is fully automated, based on the model's own token-level confidence, with no human in that loop. This is a different concern from ordering already-retrieved passages within a single prompt (a lost-in-the-middle style problem) and from a coarser one-time decision about whether to retrieve at all, since FLARE's contribution is deciding when and how often to retrieve again mid-generation.

Source: Jiang, Xu, Gao, Sun, Liu, Dwivedi-Yu, Yang, Callan & Neubig, 'Active Retrieval Augmented Generation' (2023), arXiv:2305.06983

Correct answer: .

SPLADE's sparse vector has one dimension per vocabulary term, and its explicit sparsity regularization together with a log-saturation effect on term weights pushes most dimensions to zero while letting a transformer assign nonzero weight to terms it predicts are relevant to the text's meaning, even terms that never literally appear in it; that learned term expansion is exactly what lets a document be matched on a query term it does not literally contain, addressing a vocabulary-mismatch failure that plain BM25, which only ever weights terms actually present in the text, cannot fix on its own. Because the representation stays genuinely sparse, it can still be served through the same efficient inverted-index infrastructure BM25 relies on, rather than requiring a dense approximate-nearest-neighbor index. The option describing dimensions as document IDs is wrong because SPLADE's dimensions are vocabulary terms, not other documents. The option describing a single nonzero dimension is wrong because SPLADE produces many nonzero term weights per vector, not just one. The option describing the vector as a dense, uninterpretable float array is wrong because SPLADE's whole design point is that its dimensions remain interpretable as specific vocabulary terms and the vector itself is sparse in content, not merely in storage format.

Source: Formal, Piwowarski & Clinchant, 'SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking' (2021), arXiv:2107.05720

Correct answer: .

Matryoshka Representation Learning trains a single model so that information is encoded coarse-to-fine across the vector's dimensions, explicitly optimizing many nested prefixes of the full vector, not just the complete vector, to each stand on their own as a usable representation; because of this training objective, simply keeping the first 256 of 1536 dimensions still yields a meaningful, independently useful embedding, whereas truncating a conventionally-trained model's vector would discard dimensions that were never individually optimized to carry a self-contained representation and would badly damage quality. The paper reports large storage and speed gains (up to roughly 14x smaller size and 14x faster large-scale retrieval at comparable accuracy) with no additional cost imposed at inference time. The option describing two independently trained models is wrong because Matryoshka Representation Learning produces one model whose single training run yields many usable prefix lengths, not two separate models. The option describing the later dimensions as deliberately pure noise is wrong because those dimensions still carry real, useful fine-grained information; they simply are not required for a coarser but still meaningful representation. The option restricting truncation to only the query side is wrong because the technique is meant to shrink stored vectors on both the query and document sides symmetrically, which is exactly what produces the storage and speed savings.

Source: Kusupati, Bhatt, Rege, Wallingford, Sinha, Ramanujan, Howard-Snyder, Chen, Kakade, Jain & Farhadi, 'Matryoshka Representation Learning' (2022), arXiv:2205.13147

Correct answer: .

SBERT's guidance distinguishes asymmetric semantic search, where the query and the matching text differ substantially in length and character such that swapping them would not make sense (a short question against a long reference passage), from symmetric semantic search, where query and matching text are comparable in length and structure such that swapping them would still make sense (one question against a database of other questions); because a model can be pre-trained with either case's structure in mind, using a model built for one case on data shaped like the other can misalign the resulting embeddings and quietly degrade search quality even though the pipeline still runs without errors. This is a different failure mode from mixing embeddings produced by two different model versions or checkpoints within the same index (a versioning-compatibility problem); this is instead a design decision about choosing the right category of pre-trained model for the shape of the task before any index is ever built. The option describing a language restriction is wrong because the symmetric/asymmetric distinction is about query-versus-document length and structure, not language. The option claiming scenario 2 cannot use embeddings at all is wrong because duplicate-question detection is a standard symmetric semantic search use case. The option claiming cosine similarity erases any difference between the scenarios is wrong because the metric used for comparison does not fix a mismatch between what the underlying embedding model was actually trained to represent well.

Source: Sentence-Transformers (SBERT), 'Semantic Search' documentation, https://sbert.net/examples/sentence_transformer/applications/semantic-search/README.html

Correct answer: .

Late chunking reverses the conventional pipeline: instead of splitting the raw text into chunks and embedding each one independently, it first passes the entire document through a long-context embedding model in one pass, producing contextualized token-level embeddings that already reflect the whole document. Only after that does it apply chunk boundaries and pool the token embeddings within each boundary into a single per-chunk vector. Because the pooling happens after the whole-document encoding, a chunk mentioning only 'the company' still ends up with a vector shaped by the specific company named earlier in the document, which plain chunk-then-embed cannot capture. Training a new model per document is wrong because late chunking reuses one existing long-context model without any retraining. Skipping chunking entirely and doing it inside the prompt is wrong because late chunking still produces separate stored chunk vectors for indexing, it just defers the pooling step. Prepending an LLM-generated summary to each chunk before embedding describes a different, textual technique (generating explicit context text per chunk) rather than late chunking's approach of deferring pooling on the embeddings themselves.

Source: Jina AI, 'Late Chunking in Long-Context Embedding Models' (jina.ai/news); Günther et al., 'Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models' (arXiv:2409.04701)

Correct answer: .

RAG-Fusion's pipeline has three stages: an LLM first diversifies the original query into multiple differently-worded versions of the same underlying question; each of those versions is used to retrieve its own independently ranked list of passages; and those separate ranked lists are then combined, typically with reciprocal rank fusion, into one final ranking. This directly attacks the vocabulary-mismatch problem, since a passage phrased in a way the original query never matched can still be found by one of the generated variations, while passages that rank well across several variations are reinforced rather than left to depend on a single phrasing. Training a new embedding model per query is wrong because RAG-Fusion changes the query side of the pipeline, not the embedding model itself, and needs no retraining. Simply retrieving more passages for the one original query is wrong because it does nothing to fix a genuine vocabulary mismatch; a differently-worded passage can still rank outside even a much larger single-query result set. Translating into other languages addresses a cross-lingual gap, not the same-language rewording problem RAG-Fusion targets.

Source: Rackauckas, 'RAG-Fusion: a New Take on Retrieval-Augmented Generation' (arXiv:2402.03367)

Correct answer: .

Binary quantization thresholds every embedding dimension to a single bit, which lets Hamming distance be computed extremely cheaply (a few CPU cycles per comparison) but throws away most of the fine-grained magnitude information the original floating-point vector carried, costing retrieval accuracy. The standard fix, following the rescoring approach shown to preserve roughly the bulk of retrieval performance, is a two-stage search: use the cheap binary comparison across the entire index to quickly produce a small candidate shortlist, then rescore only that shortlist using the original, higher-precision embeddings (float32 or int8) to recover an accurate final ranking. This keeps both the memory savings and the speed advantage of binary vectors for the bulk of the index, while paying the cost of precise comparison only on a tiny fraction of candidates. Retraining the embedding model from scratch on binary labels is not how existing binary-quantization pipelines work; they quantize embeddings a general-purpose model already produces. Falling back to brute-force full search for long queries doesn't address the accuracy loss from binarization itself. Simply doubling to two bits per dimension does not, by itself, restore full-precision accuracy without an added rescoring step.

Source: Sentence-Transformers, 'Embedding Quantization' documentation (sbert.net); Cohere / Yamada et al. (2021) binary-embedding rescoring approach

Correct answer: .

E5 is described as a strictly asymmetric model that breaks the query/document symmetry using two prefix identifiers rather than two separate networks: text intended as a search query is prepended with 'query: ' and text intended as a candidate document is prepended with 'passage: ' before either is tokenized and embedded, because that is exactly the convention the model was trained under. Because the model learned to represent the two roles differently based on this prefix, omitting the prefix, using the wrong one, or swapping which text gets which prefix departs from the training distribution and produces a measurable drop in retrieval quality, even though nothing about the pipeline errors out or looks obviously broken. The option claiming prefixes are ignored is wrong because E5's guidance and documented training setup both treat the prefix as load-bearing, not cosmetic. The option describing the prefix as a metadata filter is wrong because it is part of the text fed into the embedding model itself, not a separate filtering mechanism. The option claiming E5 is symmetric and uses one shared prefix is wrong because E5 is explicitly asymmetric, with query and passage treated as distinct roles by design.

Source: intfloat/e5-large-v2 model card (Hugging Face); Wang et al., 'Text Embeddings by Weakly-Supervised Contrastive Pre-training' (arXiv:2212.03533)

Correct answer: .

RAGAS's context precision metric specifically evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones for a given query, rather than merely checking whether relevant chunks are present anywhere in the retrieved set. It is computed in a ranking-aware way: an LLM judge assigns a relevant/not-relevant verdict to each retrieved passage at its actual position in the list, and those position-by-position verdicts are combined using an Average-Precision-style calculation, so a relevant passage near the top of the ranking contributes much more to the score than the same relevant passage buried near the bottom, and an irrelevant passage sitting at the very top can substantially depress the score. Context recall is a different metric that checks whether the retrieved set as a whole covers the reference answer's claims, without caring about order, so it would not penalize burying a good passage at the bottom. Faithfulness checks whether the generated answer's claims are supported by the retrieved passages, which is unrelated to how those passages were ranked. Plain set-based recall@k also ignores ranking entirely, which is exactly the gap RAGAS's context precision is designed to close.

Source: RAGAS documentation, 'Context Precision' (docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)

Correct answer: .

A contextual-compression retriever wraps an existing base retriever and, after that retriever returns its documents, runs each one through an LLM-based extractor that pulls out only the statements genuinely relevant to the query and drops the rest of that document's text, rather than passing the full, mostly-irrelevant document through to generation. This shrinks what actually reaches the prompt and removes noise that could otherwise distract the generator or waste context budget, while still preserving the same set of source documents the base retriever chose. Re-embedding and re-ranking with a smaller model changes which documents are ordered where, but does nothing to trim irrelevant text out of the documents that are kept, so it does not solve the described problem. Fetching additional documents adds more material rather than compressing what was already retrieved, and does not address the specific complaint that each retrieved document is mostly noise. Deleting documents from the underlying index is a permanent, corpus-wide change unrelated to compressing a single query's retrieved results, and would remove those documents from consideration for every future query too.

Source: LangChain documentation, 'Improving Document Retrieval with Contextual Compression' (langchain.com/blog); LangChain ContextualCompressionRetriever / LLMChainExtractor reference

Correct answer: .

IRCoT interleaves retrieval with chain-of-thought reasoning specifically because what needs to be retrieved next depends on what has already been derived, which in turn may depend on what was retrieved before: it generates one reasoning sentence at a time, and each newly generated sentence is used as the retrieval query for the next step, so once an intermediate fact (such as a specific person's identity) appears in the reasoning, the very next retrieval can be grounded in that fact rather than only in the original question. This lets later hops find passages the original question's wording alone could never have surfaced. Refusing any retrieval after the first pass is exactly the one-step 'retrieve-and-read' limitation IRCoT was built to overcome. Fine-tuning the model to memorize the fact chain removes the point of retrieval-augmented reasoning altogether and doesn't generalize past the specific facts trained on. Retrieving an unusually large number of passages up front still relies on the original question to formulate every one of those retrieval queries, so it cannot retrieve a passage keyed on an intermediate fact that hadn't yet been identified when that single retrieval pass ran.

Source: Trivedi et al., 'Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions' (ACL 2023, arXiv:2212.10509)

Correct answer: .

RankGPT prompts an instruction-following language model to output a full permutation of candidate identifiers ordered by relevance rather than scoring passages one at a time, but since a large candidate list typically cannot fit in a single prompt alongside the query, it processes the list through a sliding window: it ranks the candidates in one window, carries the top-ranked survivors of that window forward into the next window together with a fresh batch of candidates, and slides forward through the whole list this way, progressively bubbling the most relevant candidates toward the front without ever needing to fit every candidate into one prompt simultaneously. Randomly sampling and permanently discarding the rest of the candidates would silently drop potentially relevant passages rather than ranking the full list. Using a smaller cross-encoder to pre-score everything and only ever showing the LLM one candidate at a time abandons RankGPT's core idea of comparing multiple candidates against each other listwise in a single prompt. Truncating passage text so the entire list fits in one prompt regardless of length is not how RankGPT handles scale; it processes the list in windows instead of cramming everything into one call.

Source: Sun et al., 'Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents' (RankGPT paper); RankGPT sliding-window reranking documentation

Correct answer: .

CLIP trains a separate image encoder and a separate text encoder jointly, using contrastive learning that pulls a matched image-text pair's two embeddings close together in a single shared vector space while pushing mismatched pairs' embeddings apart. Because both modalities land in that same space, a text query's embedding can be compared directly against stored image embeddings using an ordinary similarity measure like cosine similarity, letting a written description retrieve matching photographs without any intermediate translation step. Converting every image to a caption via OCR and only ever doing text-to-text search describes a completely different, caption-based pipeline, not what CLIP's own dual encoders do, and would fail on images with no readable text at all. Requiring separate indexes per modality with no cross-modal comparison contradicts the entire point of CLIP's shared embedding space, which exists precisely so text and image embeddings can be compared against each other. CLIP explicitly trains a text encoder as well as an image encoder, so text queries never need to be manually converted into example images first.

Source: Radford et al., 'Learning Transferable Visual Models From Natural Language Supervision' (CLIP paper, OpenAI, arXiv:2103.00020)

Correct answer: .

Structure-aware chunking treats a document's own structural markers, such as Markdown heading levels, as the primary chunk boundaries rather than a fixed character count or an embedding-similarity breakpoint computed after the fact. Because a well-structured Markdown document's author has already grouped semantically related content under each heading, splitting along those existing boundaries tends to produce chunks that correspond to coherent, author-intended sections, and each chunk can retain its heading path as metadata for later use. This differs from semantic chunking, which derives its breakpoints purely from computed sentence-to-sentence embedding similarity rather than from any markers the author actually wrote into the document, so describing structure-aware chunking as 'exactly like semantic chunking' misses that key distinction. Reducing every document to exactly two chunks based on a single global similarity minimum is not how either approach works and would badly fragment longer documents into oversized pieces. Converting to rigid 512-token chunks while discarding the heading markers is the opposite of structure-aware chunking, since it throws away the very structure the approach is designed to exploit.

Source: LangChain documentation, 'How to split Markdown by headers' / text splitter integrations (docs.langchain.com); Pinecone, 'Chunking Strategies for LLM Applications'

Correct answer: .

Semantic similarity in embedding space does not guarantee factual relevance. A query about a specific technical parameter might retrieve chunks discussing the same general topic (high cosine similarity) but none containing the precise number, definition, or fact needed to answer correctly. The model then either hallucinates an answer or produces a vague response. Mitigations include hybrid retrieval (combining semantic search with keyword matching), re-ranking retrieved chunks, adjusting chunk size and overlap, and adding metadata filters. Embedding collapse is a real but rare training failure, not the scenario described. Context overflow would only apply if the chunks exceeded the model's limit, which five chunks rarely do. Training data leakage describes a different problem — the model answering from memorised pre-training data — and is not about retrieval quality.

Source: Lewis et al. (2020), 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks'; Anthropic RAG best practices

Correct answer: .

Chunk size is one of the most impactful design decisions in a RAG pipeline. Small chunks produce embeddings that are tightly focused on a specific point, making retrieval more precise for narrow queries — but they may split a multi-sentence explanation across chunks, losing context the model needs. Large chunks capture more surrounding context, which helps the generation model produce coherent answers, but their embeddings become diluted across multiple topics, reducing the chance of matching a specific query. The best chunk size depends on the nature of the content and queries; many systems use overlapping chunks or hierarchical retrieval to balance this tradeoff. The option claiming smaller is always better ignores the context-loss problem. The option claiming larger is always better ignores embedding dilution. The option dismissing chunk size entirely contradicts extensive RAG engineering literature.

Source: LangChain documentation, 'Text Splitters'; Anthropic, RAG best practices guide

Correct answer: .

LangChain's recursive splitter works through its separator list from the largest structural unit to the smallest, attempting to split on paragraph breaks first because paragraphs are usually the most semantically coherent unit; whenever a piece produced at one level is still larger than the target chunk size, the splitter recurses on that oversized piece using the next separator in the list (newlines, then spaces, then individual characters), so chunks stay close to natural boundaries whenever possible and only fall back to finer-grained cuts when necessary. The option describing similarity-based boundaries is a different technique, semantic chunking, which measures embedding distance between sentences rather than trying a fixed separator hierarchy. The option describing a language model choosing boundaries directly is not how this splitter operates; it uses no model calls at all, only string-based separator matching. The option describing a fixed-character cut followed by after-the-fact merging inverts the actual order of operations, which splits top-down by separator priority rather than cutting first and reconciling afterward.

Source: LangChain documentation, 'Split by character (recursively)' (docs.langchain.com/oss/python/integrations/splitters)

Correct answer: .

This describes pseudo-relevance feedback: rather than asking a real user to judge which results are relevant, the system simply assumes its own top-ranked documents from the first pass are relevant, and a Rocchio-style update moves the query vector toward the centroid of terms drawn from those assumed-relevant documents and away from the rest of the collection, producing an expanded query for a second pass. Its central weakness is exactly that assumption: pseudo-relevance feedback works well on average but can actively hurt performance on queries where the initial top results are already off-target, since the expansion terms then come from irrelevant material and drag the second search further from the user's intent. The option describing caching addresses reuse of prior results, not query expansion from assumed-relevant terms. The option describing fusing two ranked lists is a distinct technique for combining separate retrieval methods' outputs, not for expanding a single query using its own top results. The option describing a pairwise reranking model addresses scoring candidate passages more precisely, not generating new query terms, so its expense is unrelated to this technique's actual weakness.

Source: Manning, Raghavan & Schutze, 'Introduction to Information Retrieval', Chapter 9: Relevance feedback and query expansion (nlp.stanford.edu/IR-book/pdf/09expand.pdf)

Correct answer: .

Giving a model function-calling access to multiple distinct data sources and letting it choose which one to invoke for a given question is query routing within agentic RAG: the model treats each retrieval source as just another callable tool and picks based on the question's nature, exactly as it would pick between any two unrelated functions. This differs from Self-RAG, which decides only whether to retrieve at all from a single index using reflection tokens the model generates about its own need for external information, and from FLARE, which assumes one index and instead decides when during multi-step generation to trigger another retrieval pass from that same source. The option describing HyDE addresses generating a hypothetical passage to embed for a single search, not choosing among several distinct tools. The option describing RAG-Fusion addresses reformulating one query into several variants against one index and fusing the results, not routing across different kinds of sources. The option describing grading and possibly discarding retrieved passages after the fact addresses retrieval quality control, a separate concern from deciding which source to query in the first place.

Source: LlamaIndex documentation, 'Router Query Engine' and 'Agentic strategies' (docs.llamaindex.ai) -- query routing across multiple retrieval tools as the simplest form of agentic RAG

Correct answer: .

RAGAS computes faithfulness by first using an LLM to break the generated answer down into individual, self-contained factual claims, then separately judging each claim against the retrieved context to decide whether that specific claim is actually supported by what was retrieved; the final score is simply the proportion of claims that pass this check, so a 0.6 means about 60% of the answer's atomic claims could be traced back to the retrieved passages while the rest could not. It does not train or rely on a separate domain-specific classifier; the judgment is done per-claim by prompting an LLM, not by a single trained model producing one confidence value for the whole answer. It is not computed from embedding similarity between the answer and the passages either, since two texts can be semantically similar in vector space while still making unsupported factual claims, which is exactly the gap this claim-level check is designed to catch. It also does not depend on comparing against pre-written human reference answers; faithfulness measures groundedness in the retrieved context itself, not similarity to a separately authored gold answer.

Source: RAGAS documentation, 'Faithfulness' metric (docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness)

Correct answer: .

Semantic caching stores past queries as embeddings alongside their answers, and treats an incoming query as a cache hit whenever its embedding lands close enough to one already stored, which is exactly why differently worded but equivalent questions like the two password examples can share one cached answer without triggering retrieval or generation again. The option describing prompt caching addresses a different problem: reusing the computation for a literal, repeated block of prompt text such as a fixed system prompt across calls, keyed on exact text reuse rather than paraphrase-tolerant similarity between different users' independent questions. The option describing extending context via summarizing older turns addresses conversation length management, not deduplicating semantically similar independent queries. The option describing extracting only relevant sentences from retrieved passages addresses shrinking what gets sent to the generation step after retrieval already happened, not avoiding the retrieval-and-generation cycle entirely for a repeat question.

Source: Zilliz / GPTCache documentation, 'What is GPTCache' -- semantic caching for LLM and RAG applications (gptcache.readthedocs.io)

Correct answer: .

These deliberately-chosen, superficially-similar-but-wrong passages are called hard negatives, and including them in contrastive fine-tuning (using a loss such as triplet loss or a multiple negatives ranking loss) forces the model to actively separate the truly relevant passage from close look-alikes rather than just from obviously unrelated text, which is exactly the fine-grained distinction a domain like law full of similar boilerplate needs; plain random negatives sampled from the whole corpus would almost never be this close, so the model would never be pushed to learn the distinction. Treating them as partially correct 'soft positives' would defeat the purpose, since the whole point is to teach the model they are wrong despite looking similar. Calling them anchor documents describes something unrelated to contrastive training with hard negatives; an anchor in this kind of training is typically the query or the positive example, not the negative. Calling this a distillation setup misdescribes the method entirely: distillation copies a teacher model's outputs, whereas hard-negative fine-tuning is a supervised contrastive training procedure using labeled relevance judgments, not another model's predictions.

Source: Sentence-Transformers (SBERT) documentation, 'Losses' and hard negative mining guidance for fine-tuning embedding models (sbert.net/docs/package_reference/sentence_transformer/losses.html)

Correct answer: .

Anthropic's Citations feature works by having Claude chunk the documents supplied in the request and split its own response into text blocks, each carrying citations that reference specific locations in those source documents, such as character ranges for plain text or page numbers for PDFs; critically, the cited text itself is extracted directly from the source document rather than generated by the model, which is what guarantees the citation reflects real source content instead of a plausible-sounding fabrication. The option describing Claude inventing references from general training knowledge is the opposite of how this feature works, since citations are grounded in the actual supplied documents. The option describing a separate post-hoc fact-checking call misdescribes the mechanism, which is built into the single response generation step rather than a second matching pass afterward. The option describing developer-supplied manual tags is unnecessary; the chunking and citation mapping happen automatically without requiring the developer to pre-tag any source sentences.

Source: Anthropic, 'Citations' documentation (platform.claude.com/docs/en/build-with-claude/citations)

Correct answer: .

Multilingual embedding models like multilingual-E5 are trained on large sets of translation pairs across many languages so that pieces of text expressing the same meaning end up close together in one shared vector space regardless of which language they were written in; this means a French query and an English passage about the same concept land near each other, letting ordinary similarity search retrieve across languages directly with no translation step at query time. This is different from doing machine translation as a hidden preprocessing step, which would require an entirely separate translation model and a second, monolingual embedding pass rather than one shared embedding space. It is also different from maintaining separate per-language spaces linked by a manual word-to-word lookup table, which would break down for any phrase or concept not captured by single-word mappings. And it is not the case that multilingual retrieval is impossible without building duplicate indexes; the entire purpose of a shared multilingual embedding space is to avoid exactly that duplication.

Source: Wang et al., 'Multilingual E5 Text Embeddings: A Technical Report' (arXiv, Microsoft); Elastic, 'Multilingual vector search with the E5 embedding model'

Correct answer: .

The real fix is to measure and split chunks in the exact unit the embedding model enforces, which is tokens, not characters, typically using the same tokenizer the model itself uses (such as a tiktoken-based counter for many popular models); the ratio of characters to tokens is not constant, and code, non-English text, and unusual punctuation routinely produce more tokens per character than plain English prose, so a character-count target that looks safely under the limit for typical English text can quietly exceed the token limit for other content. Simply lowering the character target treats the symptom without fixing the underlying mismatch, since the same variable characters-per-token ratio still applies and could still push some chunks over the limit while unnecessarily shrinking others. Assuming one word always equals one token is incorrect; tokenizers routinely split uncommon words, code identifiers, and non-English words into multiple subword tokens. And the model's token limit is a fixed property of how the model was trained and served, not a client-side setting that can simply be raised in application code.

Source: LangChain documentation, 'Split by tokens' (docs.langchain.com/oss/python/integrations/splitters/split_by_token) -- tiktoken-based token splitting vs. character-based splitting

Correct answer: .

The recommended approach for large tables is to split by row instead of by a fixed character count, while repeating the header row or otherwise attaching the relevant column labels to every resulting row-level chunk, so a chunk retrieved on its own still carries enough context, such as which column each value belongs to, to be interpretable rather than becoming a meaningless string of numbers or labels. Discarding tables entirely throws away real information the system could otherwise answer questions from, which is a much larger loss than the chunking problem it avoids. Keeping every table as one single chunk regardless of size ignores the fact that very large tables may still need to be split to fit reasonably within a chunk size budget; the fix is to split more carefully, not to refuse to split at all. Converting the table into an image and relying on captioning at query time discards the tabular data's actual structured values in favor of a lossy visual description, which is a different, much coarser approach than preserving the row-level text data directly.

Source: Oracle, 'RAG Chunking and Parsing for Tables, PDFs, Transcripts, and Media' (blogs.oracle.com/developers) -- row-level table chunking with repeated header context

Correct answer: .

Scalar quantization maps each float32 dimension onto one of 256 discrete int8 levels using a linear transformation calibrated per collection, typically to the range covering roughly the middle 99% of observed values so the rare extreme outliers don't stretch the mapping and waste most of the available levels on values that barely occur. Because each dimension keeps 256 distinct levels of magnitude information instead of being collapsed to a single positive-or-negative bit, the resulting vectors preserve far more of the original embedding's structure than binary quantization does, which is why benchmarks consistently show scalar quantization landing within a fraction of a percent of full-precision retrieval accuracy while binary quantization costs noticeably more. The trade is a smaller compression ratio: roughly a 4x memory reduction (float32 to int8) rather than binary quantization's much larger reduction from collapsing every dimension to one bit. The option describing a larger reduction than binary quantization is backwards, since scalar quantization's byte-per-dimension representation is larger than binary's bit-per-dimension one; it also incorrectly claims dimension count itself is reduced, which quantization does not do. Neither quantization method requires retraining the embedding model; both are calibrated after the fact from the vectors a pretrained model already produced. And the two methods are not the same representation reused with a different distance function: binary quantization discards magnitude information down to a sign bit, while scalar quantization retains a much finer-grained int8 approximation of that magnitude, which is exactly why their accuracy profiles differ.

Source: Qdrant, 'Scalar Quantization for Vector Search' (qdrant.tech/articles/scalar-quantization/); cross-referenced against Qdrant's quantization overview comparing scalar, product, and binary methods

Correct answer: .

A follow-up message like 'What about for furniture?' only makes sense in light of the conversation that came before it, and a vector index has no way to recover that missing context from the follow-up's own wording alone, so embedding it as-is against the index tends to retrieve passages about furniture in general rather than about a furniture refund window specifically. The standard fix, sometimes called query contextualization or question condensing, uses a dedicated LLM call with the chat history and the new message as input, prompted to produce a single self-contained question that restates whatever the follow-up was implicitly relying on the history to supply; only that rewritten, standalone question is then embedded and sent to the retriever, while the answer-generation step afterward still has access to the full conversation. Always retrieving using only the first message would ignore that the conversation's topic can genuinely shift across turns, and would miss a follow-up asking about something new. Concatenating the entire raw history into the query is a much cruder approach that both grows unboundedly with a long conversation and dilutes the embedding with irrelevant earlier turns rather than producing one focused, self-contained question. Skipping retrieval for short follow-ups would still fail to answer questions like this one that plainly need document lookup, and conflates message length with whether retrieval is needed.

Source: LangChain documentation, 'Conversational RAG' / 'Build a Retrieval Augmented Generation (RAG) App: Part 2', create_history_aware_retriever (python.langchain.com)

Correct answer: .

efSearch controls the size of the candidate list explored while answering a single query against an already-built graph, so increasing it trades query latency for recall entirely at query time, with no need to touch or rebuild the existing index at all -- exactly the lever the team needs for improving recall on their current index. efConstruction, by contrast, controls how many candidate neighbors are explored while inserting each vector during index construction; raising it produces a higher-quality graph with better achievable recall, but that benefit only applies to vectors inserted while the higher value is in effect, so it matters for a future rebuild, not for tuning an index that already exists, and it costs slower, more memory-intensive construction rather than slower queries. The option describing efConstruction as the query-time setting and efSearch as fixed at build time has the two swapped. The option claiming they are the same setting under two names ignores that one governs one-time graph construction and the other governs every individual query, with different cost profiles. The option claiming efConstruction always beats efSearch for a fixed compute budget is not a general property of HNSW; the two parameters trade off against different costs (build time and memory versus per-query latency) and are not directly interchangeable that way.

Source: Malkov & Yashunin, 'Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs,' IEEE TPAMI 2018 (arXiv:1603.09320); cross-referenced against vector database HNSW parameter documentation (e.g. Qdrant, pgvector)

Correct answer: .

Answer correctness is the RAGAS metric built specifically to compare a generated answer against a ground-truth reference answer, which is exactly the input the other three metrics named here don't need: faithfulness only checks the generated answer's claims against the retrieved passages, context recall and the other retrieval-side metrics only look at the retrieved passages, and answer relevancy only checks how directly the generated answer addresses the question, none of them requiring a human-written reference at all. Answer correctness combines two components: a factual similarity score derived from treating statements in the answer and the ground truth as true positives, false positives, and false negatives and computing an F1-style overlap, and a semantic similarity score computed from the cosine similarity between the generated answer's and ground truth's embeddings, combined into a single weighted score. Context precision is a real RAGAS metric and does concern ranking order, but that has nothing to do with needing a reference answer; it's a retrieval-side metric computed from the retrieved passages alone. Faithfulness's defining feature is checking claims against retrieved context, not against a reference answer, so redefining it that way describes a different metric, not a variant of faithfulness. Answer relevancy measures how directly an answer addresses the question's intent using the question and answer alone, not by comparing to any reference.

Source: RAGAS documentation, 'Answer Correctness' (docs.ragas.io/en/v0.1.21/concepts/metrics/answer_correctness.html); Es et al., 'RAGAS: Automated Evaluation of Retrieval Augmented Generation,' arXiv:2309.15217

Correct answer: .

RAGAS's test set generation takes the team's own documents as input and uses an LLM to generate the questions, the supporting context, and the reference (ground-truth) answers together, rather than requiring any of those three to be hand-written first, which is what makes it a genuine shortcut around weeks of manual labeling. Rather than only generating simple, direct factual questions, it applies what its documentation calls an evolutionary generation process, deliberately transforming straightforward questions into more complex variants: ones requiring multi-step reasoning, ones with added conditions attached, and ones whose answers require synthesizing multiple passages rather than just one, so the resulting evaluation set doesn't skew toward only the easiest possible questions a pipeline might face. Replaying production queries and treating whatever a retriever happened to return as automatically correct would just encode the retriever's own existing mistakes as ground truth rather than producing an independent check on quality. Downloading a fixed, generic benchmark unrelated to the team's own documents would evaluate the pipeline against the wrong corpus entirely, missing whatever is specific about their content and users' actual questions. And requiring reference answers to be hand-written first would defeat the entire point of the feature, which is to automate generating all three parts, not just two of them, from source documents alone.

Source: RAGAS documentation, 'Synthetic Test Data Generation' / 'Generate a Synthetic Test Set' (docs.ragas.io/en/v0.1.21/concepts/testset_generation.html and getstarted/testset_generation.html)

Correct answer: .

Explicitly instructing the generator to answer only from the supplied passages and to state directly when those passages don't contain the needed information gives the model a permitted, expected output for exactly the failure case described, instead of leaving 'always produce a confident, complete-sounding answer' as the only behavior the prompt implicitly asks for; this kind of explicit-refusal instruction is documented as one of the most direct levers for reducing this particular failure mode, sometimes reinforced further by asking the model to first quote or extract the supporting passages before answering, which grounds the response in the actual retrieved text. Lowering temperature to zero makes output more deterministic but does not itself teach the model when to decline to answer; a deterministic model can still confidently generate an unsupported answer every single time if nothing in the prompt tells it that declining is an option. Retrieving more passages doesn't help on a query where the needed information is genuinely absent from the corpus altogether; more irrelevant passages just add noise rather than covering the same gap. Dropping retrieved passages for short questions abandons retrieval altogether for an arbitrary subset of queries and doesn't address the underlying prompting gap that causes overconfident answers when passages are present but insufficient.

Source: Anthropic documentation, 'Reduce hallucinations' and 'Long context tips' (docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/reduce-hallucinations; docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/long-context-tips)

Correct answer: .

MTEB was built precisely because a single benchmark like semantic textual similarity does not reliably predict performance on a different kind of task, such as retrieval, where the goal is finding a passage that specifically answers a query rather than just being semantically close to it in a general sense; the paper introducing MTEB reports that no single embedding model tested dominates across all of its task categories, directly undercutting the assumption that a leaderboard-topping score on one task transfers cleanly to another. MTEB spans eight distinct task categories, including bitext mining, classification, clustering, pair classification, reranking, retrieval, semantic textual similarity, and summarization, specifically so that a model can be compared on the kind of task closest to a team's actual use case rather than only on STS. It does not replace prior benchmarks with a single new image-text task; MTEB's original scope is text embedding evaluation across the many task types listed above. It is not simply a bigger pile of STS datasets alone; STS is only one of its several distinct task categories. And it does not collapse everything into one overall score that guarantees the top-ranked model is best for every use case; the paper's own findings argue against exactly that assumption, since per-task rankings can and do diverge from the overall average.

Source: Muennighoff, Tazi, Magne & Reimers, 'MTEB: Massive Text Embedding Benchmark,' EACL 2023 (arXiv:2210.07316)

Correct answer: .

The Dense X Retrieval paper defines a proposition as an atomic expression of a single distinct factoid, written as a concise, self-contained natural-language statement, and finer-grained than either a passage or even a full sentence. To produce these at scale, the authors first prompted a large model (GPT-4, given the proposition definition and one demonstration example) to generate propositions for tens of thousands of passages, then fine-tuned a smaller sequence-to-sequence model (a 'Propositionizer', built on Flan-T5-large) on those generated pairs so it could be run cheaply going forward; the resulting decomposition splits compound sentences into separate statements, pulls a named entity apart from descriptive information attached to it, and decontextualizes references (rewriting a pronoun like 'it' or 'the company' back into the specific entity it refers to) so each proposition can be understood in isolation. A fixed token-length cutoff is a purely mechanical split that says nothing about whether the result is a coherent, self-contained factoid, which is the actual property propositions are defined by. Propositions are explicitly a finer unit than a full sentence, not a renaming of sentence-level chunking, since a single sentence can itself bundle multiple distinct factoids that get split into separate propositions. And the Propositionizer is a trained, LLM-distilled model, not a rule-based tagger mechanically splitting at conjunctions; the paper's own methodology is about the quality of a learned decomposition, not a fixed grammatical rule.

Source: Chen, Wang, Chen, Yu, Ma, Zhao, Zhang & Yu, 'Dense X Retrieval: What Retrieval Granularity Should We Use?', EMNLP 2024 (arXiv:2312.06648)

Correct answer: .

NDCG is built specifically to combine graded relevance with position-sensitive discounting: its Discounted Cumulative Gain (DCG) component sums each retrieved result's relevance gain (typically an exponential function of the graded relevance score) divided by a logarithmic discount that grows with rank position, so a highly relevant result buried near the bottom of the list contributes much less to the total than the same result ranked at the top would. Dividing that DCG by the Ideal DCG, the maximum possible DCG obtainable by sorting the same set of results in the best possible order, normalizes the score to a 0-to-1 range that's comparable across queries with different numbers of relevant results. Recall@k and precision@k, even if naively adapted to sum or average graded scores, don't discount by position at all in their standard form, so both options describing them as position-sensitive here misrepresent the metrics as normally defined; recall and precision at k treat every result within the top k window identically regardless of exactly where in that window it lands. MRR is a real, well-established ranking metric, but it is defined around the position of the first relevant result only and doesn't incorporate graded relevance scores or every result's contribution the way NDCG's summed, discounted gain does.

Source: Järvelin & Kekäläinen, 'Cumulated Gain-Based Evaluation of IR Techniques,' ACM Transactions on Information Systems, 2002; cross-checked against 'Discounted cumulative gain,' Wikipedia

Correct answer: .

Because BM25 produces unbounded scores while a typical vector search's cosine similarity is bounded (for example between -1 and 1), directly summing the two raw scores would let whichever search happens to produce larger numbers dominate the combined ranking regardless of actual relevance; weighted fusion approaches address this by first normalizing each list's scores onto a shared scale (commonly 0 to 1) and then combining them as alpha times one list's normalized scores plus (1-alpha) times the other's, where alpha=0 relies entirely on keyword search, alpha=1 relies entirely on vector search, and alpha=0.5 weights them equally. This buys a continuously tunable, explicit knob for shifting that balance toward whichever search performs better for a given use case, which RRF's rank-only formula doesn't expose in the same direct way. RRF avoids the normalization step entirely because it only ever looks at each result's rank position within its own list, never at the raw score's magnitude, which is precisely why RRF needs no normalization while the weighted approach does. The two approaches are not mathematically identical: one operates purely on ranks, the other on normalized score magnitudes, and they can produce different final orderings. Nothing about weighted fusion requires discarding either search's results outright when both return results. And normalization is a scaling step applied to already-produced scores; it does not require retraining the embedding model to force its native output range to match BM25's.

Source: Weaviate documentation, 'Hybrid search' and 'Alpha Parameter' knowledge card (docs.weaviate.io/weaviate/concepts/search/hybrid-search; weaviate.io/learn/knowledgecards/alpha-parameter)

Correct answer: .

The attack is indirect prompt injection: unlike direct injection, where an attacker types a malicious instruction straight into the chat interface, indirect injection plants the instruction inside an external source the application later ingests and treats as retrieved context. RAG pipelines are particularly exposed because the retrieval step has no mechanism to distinguish 'this is untrusted data' from 'this is a trusted instruction' — once a chunk scores high enough on similarity to enter the prompt, the model reads it with the same effective authority as its system prompt, and the attacker never needed access to the chat interface itself, only write access to whatever store the pipeline ingests (a ticket, an uploaded file, a web page). The option describing this as the same thing as direct injection is wrong because it collapses a meaningful distinction the industry's own prompt-injection taxonomy draws: direct injection requires attacker control of the user-facing prompt, indirect injection does not. The option describing this as training data poisoning is wrong because nothing about the attack touches the embedding model's weights or training process; it exploits what happens entirely at inference time, when a document is retrieved and concatenated into a prompt, with the embedding model's parameters completely unchanged before and after the attack. The option describing this as jailbreaking via an adversarial suffix is wrong because that technique specifically depends on the attacker crafting and submitting exact token sequences directly to the model's input, which is precisely the capability an indirect attacker lacks — they only control content elsewhere that gets pulled in later.

Source: OWASP Gen AI Security Project, 'LLM01:2025 Prompt Injection,' OWASP Top 10 for LLM Applications, https://genai.owasp.org/llmrisk/llm01-prompt-injection/

Correct answer: .

A self-querying retriever is given a description of the document content plus the name, type, and description of each metadata field up front. When a query like this comes in, an LLM chain parses it into a structured query: a narrower semantic string, stripped of the filter conditions, to run through the vector similarity search, plus a separate structured filter expression built from the fields the query actually referenced, which a store-specific translator then converts into that backend's own filter syntax before the filtered similarity search runs. The option describing metadata re-embedded into the same vector space is wrong because numeric and categorical comparisons like 'less than 3' or 'equals kitchen appliances' are not semantic similarity relationships at all, and folding them into the same continuous embedding space used for meaning would distort that space rather than enforce an exact structured condition. The option describing a second LLM call manually screening every top-k result is wrong because that describes filtering after retrieval has already happened on the unfiltered query, which both wastes the top-k budget on irrelevant candidates and is a fundamentally different architecture from constructing a filter the vector store itself applies during search. The option requiring the user to hand-write filter syntax is wrong because it describes the opposite of what makes this retriever useful: the entire point is translating a natural-language question into that structured filter automatically, without ever asking the user to know or write the store's native query language.

Source: LangChain documentation, "How to do 'self-querying' retrieval," https://js.langchain.com/v0.2/docs/how_to/self_query/

Correct answer: .

The two setups differ in what kind of judgment they produce. Each RAGAS metric scores one answer in isolation: faithfulness checks whether the generated claims are supported by the retrieved passages, context recall checks whether the retrieved passages cover what's needed, and answer relevancy checks whether the answer addresses the question, so every test question yields an absolute number for a single pipeline run. A pairwise LLM-as-a-judge setup instead shows the judge two specific answers to the same question and asks only which one is better; that single judgment carries no absolute quality information and doesn't transfer to a different pair of answers. Because of that, a one-off pairwise win doesn't answer whether the new prompt is better overall — the team needs many such pairwise judgments across the test set, then has to aggregate the resulting win and loss counts into an overall ranking, which the work behind MT-Bench and Chatbot Arena does by fitting a Bradley-Terry model to the collected pairwise outcomes, the same method underlying the Chatbot Arena leaderboard, rather than by treating a few individual wins as a verdict. The option claiming the two setups are functionally identical is wrong because it ignores this absolute-versus-relative distinction entirely: an LLM judge being used somewhere in both pipelines does not make an independent per-answer score the same kind of measurement as a head-to-head preference. The option proposing BLEU-style string overlap is wrong because it removes the LLM judge that the described setup explicitly keeps, and string-overlap metrics are well known to correlate poorly with human judgments of answer quality regardless. The option requiring a human to make every final call is wrong because it contradicts the setup described, where the LLM itself is asked to pick the better answer; introducing a mandatory human-in-the-loop step for every judgment would defeat the purpose of using an LLM judge to scale evaluation past what manual review can cover.

Source: Zheng, L. et al., 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,' NeurIPS 2023, https://arxiv.org/abs/2306.05685

Correct answer: .

The standard fix is explicit deletion and upsert, keyed by a stable, deterministic vector ID assigned at ingestion time. When a source document is revised, writing its new content under the same ID overwrites the stale vector in place; when a document, or one of its chunks, is withdrawn, the team deletes that specific ID so it can never be retrieved again, rather than letting it linger because nothing in the pipeline tracks it. None of this works unless the team set up a stable document-to-ID mapping from the start, commonly a hierarchical scheme like 'documentId-chunkId,' because without it, finding exactly which vectors correspond to a given withdrawn document later requires an expensive, error-prone search by content rather than a direct lookup by ID. The option declaring embeddings irreversible and the problem unfixable is wrong because it confuses 'you can't recover the original text from a vector,' which is true but irrelevant, with 'you can't remove a vector from an index,' which is false since any vector store supports deletion by ID. The option proposing a full nightly rebuild is wrong on its own claim that nothing needs to be set up in advance: a full re-embed of every active document is far more expensive than a targeted delete or upsert, and still needs a definition of which documents currently count as active, sourced from somewhere. The option proposing a lower similarity threshold is wrong because similarity scores reflect semantic closeness to the query, not recency; a withdrawn chunk whose wording still closely matches the query keeps scoring just as high as it always did regardless of any threshold change, and per-age-bracket thresholds don't address the chunk still being retrievable at all within its own bracket.

Source: Pinecone, 'Create and manage vectors with metadata,' https://docs.pinecone.io/troubleshooting/create-and-manage-vectors-with-metadata

Correct answer: .

KATE retrieves in-context examples non-parametrically by semantic similarity: both the pool of labeled examples and the new input are embedded, and the nearest neighbors to the new input's embedding are pulled from the pool and inserted as that input's few-shot examples, so the examples shown to the model change from one input to the next rather than staying fixed. The original study found this similarity-based selection improved results over randomly chosen examples across a range of natural language understanding and generation tasks, and that fine-tuning the embeddings used for retrieval on data related to the task produced further gains on top of using an off-the-shelf embedding space. The option proposing random selection as the method, and as the better-performing one, is wrong on both counts: KATE is specifically a similarity-based alternative to random or fixed selection, and the paper's own finding was that similarity-based retrieval beat random selection, not the reverse. The option proposing a classifier that already needs to know the true label is wrong because it describes something circular and useless at real inference time; the entire point of in-context example selection is to pick good examples for an input whose label is not yet known, which rules out any method that requires the label as an input to the selection process itself. The option proposing exact keyword overlap is wrong because it describes a lexical matching method, not the embedding-based semantic similarity KATE actually uses, and the claim that keyword overlap always outperforms embedding similarity has no support in the cited findings, which specifically credit a semantic embedding space, and fine-tuning that space, for the performance gains observed.

Source: Liu, J. et al., 'What Makes Good In-Context Examples for GPT-3?,' https://arxiv.org/abs/2101.06804