Lesson 1 established why agents resist simple pass/fail testing: they take multi-step paths, call tools, and can reach a correct outcome through very different trajectories. This lesson zooms into the two loops every team running an agent in production actually needs to close the gap between "looks right in review" and "works for real users": offline evaluation and online monitoring. They are not two names for the same idea — they run on different data, on different cadences, and catch different classes of failure, and confusing them is one of the fastest ways to ship a regression behind a fully green test suite.
Offline evaluation runs a fixed, versioned dataset against the agent before code ships — typically in continuous integration, on every pull request or at minimum before every deploy. Because the dataset and the scoring function are both fixed, the run is fast, cheap, and perfectly repeatable: the same 50 or 500 cases produce the same score today as they will tomorrow, so a change in that score is attributable to the code change and nothing else. That repeatability is also its ceiling — the dataset can only ever be a proxy for the infinite variety of real user input, curated by humans who wrote down the failure modes they already knew about.
Online monitoring runs continuously against real production traffic after code ships. It sees inputs nobody wrote down in advance, at a scale and diversity no hand-curated set can match, but ground truth about whether the agent actually succeeded often arrives late — a user complains three days later — or never arrives at all if the user simply leaves. Because you cannot manually label every production request, monitoring leans on sampling and on guardrail metrics: proxies like escalation rate, latency, and refusal rate that are cheap to compute continuously and correlate with real failure even without a human verdict on every case.