A RAG evaluation reports high context recall, meaning the retrieved passages contain essentially all the information needed to answer the question, but low faithfulness, meaning much of the generated answer is not actually supported by those retrieved passages. What does this particular combination of scores most directly indicate is going wrong?
- The retrieval component failed to find the relevant passages, so the retrieved context is missing key information the answer needed
- The retrieval component is doing its job, since the needed information was present in what was retrieved, but the generation step is not staying grounded in that retrieved context and is producing claims the context does not actually support
- The embedding model used for retrieval is outdated and should be replaced with a newer one in order to fix the low faithfulness score
- Context recall and faithfulness measure the same underlying property from two different angles, so a high score on one should always produce a high score on the other
Why B? And why not the others?
Correct answer: B. The retrieval component is doing its job, since the needed information was present in what was retrieved, but the generation step is not staying grounded in that retrieved context and is producing claims the context does not actually support
Context recall and faithfulness are deliberately separate measurements precisely so that retrieval quality and generation quality can be diagnosed independently: a high context recall score means the necessary supporting information was present among the retrieved passages, so the retrieval half of the pipeline did what it needed to do, while a low faithfulness score means the generated answer nonetheless contains claims that are not actually supported by that same retrieved context, pointing to a generation-time grounding problem such as the model drifting from or embellishing beyond what the passages state. The option blaming missing retrieved information directly contradicts the premise that context recall is already high, meaning the needed information was in fact present in what was retrieved. The option recommending a newer embedding model misdiagnoses the fix, since embeddings affect what gets retrieved, a stage the scenario already says is working, not whether the generated text stays grounded in that retrieved material. The option claiming the two metrics always move together denies the entire reason both are reported separately; if they were redundant, reporting one score high and the other low would be impossible in the first place.
Source: Es et al., 'Ragas: Automated Evaluation of Retrieval Augmented Generation' (2023), arXiv:2309.15217