Two evaluation approaches are compared for a customer-service agent that is supposed to look up an order, check its refund eligibility, and only then issue a refund. One approach only checks whether the refund that was eventually issued (or not issued) was the correct final outcome. The other approach also inspects the full sequence of tool calls the agent made along the way -- which lookups it ran, in what order, whether it skipped the eligibility check, whether it retried unnecessarily -- regardless of whether the final outcome happened to be correct. What is the main advantage of the second approach?
- It can catch cases where the agent reached the correct final outcome by an unreliable or incorrect path, for example skipping the eligibility check but guessing right anyway, which final-outcome-only evaluation would score as a pass since it only looks at whether the end result was correct
- It removes the need to ever check the final outcome at all, since a correct sequence of tool calls always produces a correct final outcome
- It only works for agents that use exactly one tool, since evaluating a sequence of steps is impossible once more than one tool is involved
- It is guaranteed to produce a numeric score that always agrees with a human reviewer's judgment of the same run
Why A? And why not the others?
Correct answer: A. It can catch cases where the agent reached the correct final outcome by an unreliable or incorrect path, for example skipping the eligibility check but guessing right anyway, which final-outcome-only evaluation would score as a pass since it only looks at whether the end result was correct
Evaluating only the final outcome tells you whether a run ended well, but not how it got there, so an agent that skipped a required check and still happened to land on the right refund decision would score identically to one that actually verified eligibility properly; inspecting the trajectory -- the sequence of tool calls, their order, and whether required steps were actually taken -- surfaces that kind of hidden failure even when the end result looks fine, which is exactly the gap outcome-only evaluation misses. The option claiming the final outcome no longer needs checking is wrong: a correct sequence of steps still doesn't guarantee a correct outcome if, say, a tool itself returns bad data, so both levels remain useful together rather than one replacing the other. The option limiting trajectory evaluation to single-tool agents is wrong: examining an ordered sequence of steps works precisely because there are multiple tool calls to look at; it isn't blocked by having more than one. The option claiming guaranteed agreement with a human reviewer is wrong: any automated evaluation, trajectory-based or not, can still diverge from human judgment on a given run.
Source: LangChain, 'LLM Evaluation Framework: Trajectories vs. Outputs' (langchain.com/resources/llm-evaluation-framework) -- distinguishes trajectory-level evaluation of an agent's full sequence of tool calls and intermediate steps from outcome-level evaluation of only the final result