A RAG evaluation reports high faithfulness, meaning every claim in the generated answer is well supported by the retrieved passages, but low answer relevancy. The generated answer is a lengthy, fully-sourced discussion of a topic adjacent to what was asked, without ever directly addressing the specific question the user posed. What does this particular combination of scores indicate?
- The scores are contradictory and cannot both be correct at once, since an answer that is well supported by retrieved evidence must, by definition, also directly address the question being asked
- The retrieved passages must be missing the specific information the question required, which is what low answer relevancy directly measures, so the fix is to retrieve better passages
- The generated answer is grounded in the retrieved evidence (nothing in it is unsupported), but it fails to actually address what the user specifically asked, which is a distinct failure from being ungrounded and points to a generation-time problem with staying on-topic and targeted to the question rather than a problem with whether the cited material is trustworthy
- Answer relevancy is only meaningful when faithfulness is also low, so a high faithfulness score alongside a low answer relevancy score means the answer relevancy number should be disregarded entirely
Why C? And why not the others?
Correct answer: C. The generated answer is grounded in the retrieved evidence (nothing in it is unsupported), but it fails to actually address what the user specifically asked, which is a distinct failure from being ungrounded and points to a generation-time problem with staying on-topic and targeted to the question rather than a problem with whether the cited material is trustworthy
Faithfulness and answer relevancy are deliberately measured as separate properties: faithfulness checks whether the claims in the generated answer are actually backed by the retrieved context, while answer relevancy checks whether the generated answer directly addresses the specific question that was asked, and these two things can diverge, exactly as in this scenario, where every statement made is well supported by the retrieved passages (high faithfulness) yet the answer wanders into adjacent territory instead of directly answering the question posed (low answer relevancy); this combination points at a generation-time problem with staying targeted to the actual question, not a grounding problem. The claim that the two scores contradict each other misunderstands what each one measures; being well supported by evidence and being on-topic for the specific question asked are independent properties, and an answer can easily have one without the other, as this scenario demonstrates directly. Blaming missing information in the retrieved passages misattributes what low answer relevancy actually measures; it evaluates the generated answer's focus against the question, not whether the retrieved context contained the needed facts, and nothing here indicates the passages were insufficient. And there is no rule making answer relevancy meaningful only when faithfulness is low; both metrics are reported and interpreted independently precisely so that a case like this one, high on one axis and low on the other, can be diagnosed rather than discarded.
Source: Es, James, Espinosa Anke & Schockaert, 'RAGAS: Automated Evaluation of Retrieval Augmented Generation' (2023), arXiv:2309.15217