Trajectory Evaluation: Judging the Path, Not Just the Answer

The defining problem in agent evaluation is that a correct final answer proves almost nothing about whether the agent should be trusted to run again. An agent asked to reconcile a customer refund can arrive at the exact right dollar figure after forty tool calls, one of which deleted a production record it was never supposed to touch, and cost four dollars in token spend to stumble into the answer by trial and error. Grading only pass/fail on the final answer — the outcome-focused approach the earlier lessons in this course built up — is blind to all of that; it scores this run identically to an agent that reached the same number in six clean, well-reasoned steps.

A trajectory is the ordered sequence of everything an agent produced during a single run, from its first thought to its terminal message: each reasoning step, each tool call together with its arguments, each tool result returned, and each message sent to the user or to another agent, kept in the exact order it actually happened. Trajectory evaluation asks a different question than outcome evaluation asks. Outcome evaluation asks whether the answer was right. Trajectory evaluation asks whether the process that produced it is one you would let run unsupervised again tomorrow, on a harder input, with real money or real data on the line.

This lesson treats a trajectory as a logical object: an ordered list of steps you can inspect, score against rules, and compare, independent of how it happens to be captured. The instrumentation side, where those same steps show up as spans and traces inside a tracing system, is its own lesson later in this course. This lesson also stays one level above any single step: whether one particular tool call picked the right tool and filled in correct arguments is a narrower, mechanical check, covered next — this lesson is about judging the sequence as a whole, not any individual link in it.

Analogy🏏Cricket

🏏 Think of it like cricket: imagine a domestic T20 match where a team is set 180 to win in Bengaluru and reaches it in the final over. The scoreboard alone says "target achieved" — but a coach reviewing the footage sees something different. The No. 4 batter walked out and slogged from ball one against the second over, got dropped twice at deep midwicket, ran two needless quick singles that could easily have ended in a run-out, and only found the winning runs because the rival captain panicked and threw the ball to a part-time spinner in the 19th over. The scoreboard — the final answer, target met — hides every bit of that. A trajectory is the ball-by-ball commentary of how the chase actually unfolded: every shot selected, every run taken, every risk absorbed, in order, from the first ball to the winning boundary. Grading only the final total is like grading only whether the team won; it says nothing about whether the batting was disciplined, whether the required strike rotation happened, or whether a reckless heave that could have ended the innings should ever be repeated. The insight is that a match — like an agent run — can end in a win while the path to it was one dropped catch away from disaster, and evaluating only the scoreboard makes that invisible.