Capstone: Build an End-to-End Evaluation Pipeline — Agent Evaluation & Observability

A team can own a well-built eval harness, a rigorous statistical report, a tracing pipeline, and a functioning alert on guardrail metrics, and still ship a regression to every user for four hours, because none of those four systems talk to each other. The harness runs against a static eval set nobody updates after go-live. The alert fires into a channel nobody watches until a customer complains. The trace that would have shown exactly which tool call went wrong sits in a dashboard nobody opened during the incident. Each piece works in isolation, and the system as a whole still fails, because evaluation was treated as a pre-launch gate instead of a continuous loop that production feeds back into.

Closing that loop means every stage from the previous 34 lessons has to compose. The eval set from lesson 3 needs the provenance fields from lesson 15 so a new case can be traced to the incident that produced it. The harness from lesson 11 needs the deterministic, fuzzy and judge scorers from lessons 4 through 6 plumbed in behind one pluggable interface, with trajectory and tool-argument scoring from lessons 9 and 10 folded into the same result. The statistical reporting from lessons 12 and 13 has to replace every bare percentage before the CI gate from lesson 14 is trustworthy enough to block a deploy. The tracing from lessons 18 and 19 has to feed the same production sampling and guardrail alerting from lessons 24 and 25 that lesson 29 uses to turn a real incident into tomorrow's regression test.

This lesson builds that loop as a small, runnable Python project: eight modules, each importing the ones before it, ending in one orchestrator script that runs the entire pipeline against a stub support-triage agent and prints a full report. It is deliberately not a survey of the previous 34 lessons -- it is the wiring diagram that turns them into one system instead of thirty-four separate techniques, and every module composes with the ones before it exactly the way a production pipeline's modules would.

Analogy🏏Cricket

🏏 Think of it like cricket: Rahul Dravid did not become an effective head coach by running a good nets session in isolation from a good selection process. The value of the system he ran as India's head coach came from the loop connecting four things that, taken separately, are unremarkable -- the domestic season's Ranji Trophy scorecards, the National Cricket Academy's structured trials, the India A tour results, and the senior team's actual match failures. A batter who struggles against the short ball in a home Test does not just get dropped and forgotten -- that specific failure becomes a tagged weakness, fed back into what the NCA's throwdown specialists drill against in the next camp, then measured again on the next India A tour before the player is trusted at the senior level again. Just as a brilliant NCA trial means nothing if its findings never reach the senior selection committee, a brilliant eval harness means nothing if its report never reaches the engineer who can act on it. Just as Dravid's system only works because a Test match failure is traceable back to a specific, nameable weakness rather than a vague 'poor form', a production incident only strengthens an agent if it becomes a specific, nameable eval case rather than a vague postmortem note. The insight is that evaluation and observability are not two disciplines sitting next to each other -- they are one loop, and the loop is the product, not the individual stages.