Most teams ship their first agent with zero eval cases and a bad feeling about it, or with five hundred auto-generated cases nobody trusts and nobody reads when they fail. Neither gets you what you actually need, which is a small, honest snapshot of how the agent behaves on the inputs it will really see. The previous lesson established what success even means for a non-deterministic system; this lesson turns that into an artifact you can run — a file on disk, checked into your repository, that a script can load and score against.
Building that first eval set is less about tooling and more about a sequence of honest choices: where the cases come from, how many you genuinely need before you can act on the results, how to spread them across difficulty and failure mode so a passing score means something, and how to write down what 'correct' means for a case where there is no single correct sentence. Get those choices right on a set of twenty to fifty cases and everything later in this course — deterministic checks, LLM-as-judge scoring, the full CI harness — has solid ground to stand on. Get them wrong and no amount of later tooling fixes a foundation of cases nobody trusts.
This is an exercise lesson: every code block below is something you should actually run, in order, in the same directory, so the JSONL file one script writes is the file the next script reads. By the end you will have a real `eval_set_v1.jsonl`, a loader that validates it, a versioning check that catches silent edits, and a harness that reports pass rates broken down by tag.
Analogy🏏Cricket
🏏 Think of it like cricket: Before India tour Australia, Virat Kohli does not spend three hours in the nets facing a bowling machine set to generic medium pace. The throwdown specialists at the WACA are briefed to reproduce exactly what he will face in the Test: Mitchell Starc's inswinging yorker at the death, Pat Cummins' short ball angled into the ribs, Nathan Lyon's off-break drifting into the rough outside leg stump. Kohli would rather face fifteen deliveries that mirror the real attack than five hundred throwdowns that never occur in a match situation. Each delivery earns its place in the session because it tests something specific — can he leave the wide one outside off, can he get inside the line of the yorker, can he pick the doosra out of the hand. A throwdown that resembles nothing Starc or Cummins actually bowls teaches Kohli nothing about Perth; it just burns practice time he does not have. This is the exact discipline behind a first agent eval set. A handful of cases pulled from what real users actually typed, each one chosen because it probes a distinct way the agent could fail, teaches you more in one afternoon than five hundred synthetic prompts that resemble nothing a real user would ever send. Practice that does not mirror the real contest is not preparation, it is just motion — and an eval set built from imagined cases nobody sent has the same problem.
🏏 Showing the Cricket analogy — a Cricket version isn’t available for this concept yet.