Evaluating a chatbot that answers one question at a time is close to evaluating a classifier: you have an input, a reference output, and a scoring function, and you run it over a held-out set. Evaluating an agent breaks every part of that setup. An agent takes a goal, not a single question, and reaches it through a sequence of decisions it makes on the fly: which tool to call, what arguments to pass, whether to retry, when to stop. Two runs given the identical instruction can take different numbers of steps, call different tools in a different order, and still both be correct — or both be wrong in ways a simple answer-matching check will never catch.
This matters because the tools most engineers reach for first — exact string match, a fixed reference answer, a single pass/fail assertion — were built for deterministic programs and single-turn model calls. Applied to an agent, they produce two failure modes at once: they reject correct runs that took a different but equally valid path, and they pass incorrect runs that stumbled into the right final string by luck. Neither failure is rare. Both compound silently until a team ships a regression it never saw coming, because the eval it trusted was answering a question — 'does the output match?' — that was never the right question for a multi-step, side-effect-producing system.
The rest of this course builds a real evaluation and observability practice for agents: offline eval sets, trajectory-aware scoring, LLM-as-judge harnesses calibrated against humans, and the OpenTelemetry-based tracing that makes production agents debuggable instead of opaque. This first lesson does none of that engineering yet. Its only job is to make the underlying problem concrete enough that the rest of the course reads as a solution rather than as ceremony: agents are non-deterministic, multi-step, side-effecting systems, and cost and latency are outcomes, not footnotes.