An evaluation framework for a retrieval-augmented generation (RAG) system reports three separate scores for a generated answer: a score for whether the retrieved passages are relevant to the question, a score for whether the answer directly addresses the question asked, and a score for whether each claim made in the generated answer is actually supported by the retrieved passages. Which of these is the 'faithfulness' (or 'groundedness') score, and what does it specifically measure?
- The relevance-of-retrieved-passages score is faithfulness, because a faithful answer is defined entirely by how relevant the retrieval step's passages were to the original question, regardless of what the generated answer actually claims
- The answer-addresses-the-question score is faithfulness, because a faithful answer is defined entirely by how directly it addresses the literal wording of the question, regardless of whether its claims are grounded in the retrieved passages
- Faithfulness cannot be measured separately from the other two scores, since retrieval relevance, answer relevance, and groundedness in retrieved passages are mathematically the same quantity computed three different ways
- The claims-supported-by-retrieved-passages score is faithfulness; it specifically measures whether the claims made in the generated answer can be traced back to and supported by the retrieved context, independent of whether that context was relevant to the question or whether the answer addresses the question asked
Why D? And why not the others?
Correct answer: D. The claims-supported-by-retrieved-passages score is faithfulness; it specifically measures whether the claims made in the generated answer can be traced back to and supported by the retrieved context, independent of whether that context was relevant to the question or whether the answer addresses the question asked
Faithfulness (also called groundedness) specifically measures whether each claim in the generated answer can be traced back to, and is supported by, the passages the retrieval step actually returned -- it is a check on whether the generator invented or distorted claims beyond what its retrieved evidence supports, and it is logically independent of the other two scores: an answer can be grounded in its context yet fail to address the question, or address the question well while retrieval itself was off-topic. The option labeling the retrieval-relevance score as faithfulness is wrong because relevance of the retrieved passages to the question says nothing about whether the generated answer's specific claims are actually supported by those passages. The option labeling the answer-relevance score as faithfulness is wrong for the same reason in reverse: directly addressing the question's wording says nothing about whether the answer's claims are grounded in retrieved evidence rather than fabricated. The option claiming all three scores are mathematically the same quantity is wrong because they are defined to capture distinct failure modes -- poor retrieval, off-topic answers, and ungrounded claims can each occur independently of the others.
Source: Es et al., 'RAGAS: Automated Evaluation of Retrieval Augmented Generation,' arXiv:2309.15217 (2023) -- defines faithfulness, answer relevance, and context relevance/precision as distinct RAG evaluation metrics