"The agent should be helpful and not annoying" is not a success definition -- it is a feeling wearing the clothes of one. It cannot tell a reviewer whether a given transcript passed, it cannot be handed to a second reviewer and produce the same verdict, and it cannot be compiled into an eval set because there is nothing in it to check. Turning a goal like that into something you can actually evaluate against is the first and most neglected step in building any agent evaluation system, and it has to happen before a single automated checker gets written.
This step matters more for agents than it did for traditional software because the system is non-deterministic: the same input can produce a different response on two separate runs, so you cannot fall back on "run it and see if it matches" the way a unit test does for a pure function. Without a precise definition of success, non-determinism looks like noise you can't reason about. With one, it becomes a distribution you can measure -- how often does the agent clear the bar, and how does that rate move release to release.
This lesson stays deliberately upstream of the checkers themselves. Deterministic assertions, fuzzy similarity scoring, and LLM-as-judge are all ways of *implementing* a check once you know what you're checking for -- later lessons cover each in depth. Here the question is narrower and comes first: given a vague product goal, how do you turn it into a precise, checkable, multi-dimensional definition of success that two people would apply the same way.