Building an Eval Harness — Agent Evaluation & Observability

Every eval technique from lesson 03 onward has run as a one-off script: load a handful of cases, call the agent, eyeball the output. That works for ten cases checked once. It falls apart the moment you have three hundred cases, three different scorer types, and a team that needs to run the same evaluation before every deploy without babysitting it. At that point the eval set and the scorers aren't the bottleneck anymore — the thing missing is a harness: a program whose only job is to run cases through an agent, score them consistently, and produce the same shape of report every time, fast enough that nobody skips it.

This lesson builds that harness end to end, in three pieces that compose into one working system. First, a case loader that reads a JSONL eval set and a Scorer interface that lets deterministic checks (lesson 05), fuzzy checks (lesson 06), and an LLM judge (lessons 07-08) plug in interchangeably. Second, a bounded concurrent runner with a disk-backed cache, wrapped around a fake, fully offline agent. Third, aggregation to overall and per-tag metrics and a JSON report writer. Every code block in this lesson is a real module that imports the one before it and runs — by the end you'll have a small but complete `eval_harness/` package sitting on disk, not a collection of unrelated snippets.

One design choice is worth flagging before we start: the agent under test in this lesson is a stub, not a live model call. That is deliberate, not a shortcut taken for the sake of the lesson. A harness that needs network access, API keys, and live spend just to execute is a harness nobody runs locally and nobody trusts in CI — and CI is exactly where lesson 14 plugs this harness in next. Building it against a fake agent first, and proving the *harness* is correct independent of any live model, is the same discipline as unit-testing business logic against a fake database before pointing it at a real one.

Analogy🏏Cricket

🏏 Think of it like cricket: before Royal Challengers Bengaluru face Mumbai Indians, the squad doesn't turn up to the Chinnaswamy nets and improvise. Head coach Andy Flower runs three lanes in parallel — Mohammed Siraj bowling yorkers in lane one, Yuzvendra Chahal bowling googlies in lane two, a throwdown specialist firing short balls in lane three — while Virat Kohli, Faf du Plessis and Glenn Maxwell rotate through all three. Each lane has its own assistant coach, and each one judges differently: the technique coach ticks boxes off a checklist (elbow position, foot placement — a deterministic rule), the feel coach grades timing and touch on a 1-10 scale by eye (a fuzzy judgment), and the senior batting consultant gives a holistic verdict on shot selection against the expected match situation (a judge call). Different methods, but every coach fills in the same scorecard format, so the notes fold into one report at the 7pm team meeting no matter which coach wrote them. Three lanes running at once, bounded by how many coaches and bowlers the ground can host, is what gets the whole squad through a full session before the light goes. And this happens before *every* fixture, identically, not as a one-time trial — a proper nets session is a repeatable system with the same lanes and same scorecards every time, which is exactly what an eval harness is for an agent.