Every eval built so far in this course asks a version of the same question: on a distribution of realistic, well-intentioned requests, does the agent do the right thing? That question has a blind spot. A production agent doesn't only receive requests from people trying to get their job done — it also receives text from tool results, retrieved documents, emails it summarizes, and web pages it fetches, any of which can be authored by someone whose entire goal is to make the agent do something it must never do. Measuring accuracy against benign traffic tells you nothing about what happens when the traffic is hostile.
This is a different kind of failure from the ones earlier lessons cover. A trajectory eval or an LLM judge is trying to catch the agent being wrong by accident — a bad tool choice, a hallucinated fact, an inefficient path to a correct answer. Adversarial evaluation is trying to catch the agent being wrong on purpose, because someone engineered an input specifically to produce that failure. The skill this lesson teaches is closer to security engineering than to product QA: you are not asking 'does this usually work,' you are asking 'can this be broken, and how would I know.'
That framing matters because it changes what 'good' looks like. A benign eval set you can grow organically from real user traffic. An adversarial eval set has to be built deliberately, because attackers do not show up in your logs until after they have already succeeded — and by the time you have absorbed the lesson, you want it sitting in a permanent regression suite, not written up in a postmortem and forgotten. This lesson covers threat modelling an agent, the parts of its attack surface that are unique to agents rather than to chatbots, how to build and score a permanent adversarial suite, what automated and human red-teaming each contribute, and the cost of defenses that the attack-success-rate number alone will never show you.