Every evaluation technique covered so far in this course — LLM-as-judge scoring, trajectory evaluation, tool-argument accuracy — assumes a single agent producing a single trajectory that a judge can read start to finish. Multi-agent systems break that assumption structurally, not just in scale. A supervisor agent decomposes a task and delegates to two or three specialist agents, each of which runs its own trajectory, calls its own tools, and hands its output to the next agent in the chain or back up to the supervisor. There is no single trajectory to score anymore; there is a graph of trajectories connected by handoffs, and the system's overall success or failure is a property of that graph, not of any one node in it.
This matters because the natural instinct — evaluate each agent the way you'd evaluate a single agent, then roll the scores up — produces dashboards that are individually accurate and collectively misleading. An agent can pass every per-turn check you throw at it and still be the reason the system fails, because its failure mode isn't 'gave a wrong answer' but 'gave a technically correct answer to a badly framed sub-task it was handed.' Conversely, an agent that looks broken in isolation — a high retry rate, frequent tool errors — might be catching and correcting problems that would otherwise sink the system, and shutting it down 'to improve its metrics' would make things worse.
The rest of this lesson works through four questions that per-agent evaluation cannot answer on its own: when the system fails, which agent actually caused it; how much of the failure lives in the handoffs between agents rather than inside any agent; whether the multi-agent design is even earning its cost and latency over a single well-prompted agent; and how an agent that looks perfect in isolation can still break the system it's part of.