Every evaluation technique in this course so far runs before or beside production traffic: an eval set, an LLM judge, a regression suite in CI. All of it answers “does the candidate look better on the data we already have.” None of it answers the question a launch decision actually needs answered — does the candidate perform better on the traffic you don't have yet, the traffic that shows up the moment real users, real tool outages, and real adversarial input start hitting it. An eval set that scored a candidate at 94% can still ship a regression, because the eval set is a sample, and production is the population it was sampled from imperfectly.
This lesson is about the methods that get a trustworthy answer from production itself without betting the product on a change that hasn't earned trust yet: shadow deployment, which watches a candidate on real traffic with zero user exposure; canary releases, which expose real users but in small, reversible slices with automatic rollback; and controlled experiments — A/B tests and interleaving — which get a statistically defensible answer to “is the candidate actually better,” not just “did nothing catastrophic happen.” Each method trades safety for signal in a different place, and picking the wrong one for the question you're asking is itself a common failure mode covered later in this lesson.
Agents make the experimentation half of this harder than it is for a typical web A/B test. A session is many turns, not one request, so the unit you randomize on and the moment your metric is even knowable both change. Users adapt to the agent they're given, which contaminates the very comparison you're running. And because lessons 12 and 13 already built the statistical machinery — confidence intervals for a proportion, McNemar's test for paired data, bootstrap resampling, the pass@k vs pass^k distinction — this lesson does not re-derive any of that. It applies it to a new, harder problem: choosing what to measure, when you're allowed to look, and how many sessions you need before looking at all.