Regression Testing Agents in CI — Agent Evaluation & Observability

A harness that runs when someone remembers to run it is not a regression test — it's an opinion poll with better tooling. Lesson 11 built `harness_core.py`, `harness_runner.py`, and `harness_report.py`: a case loader, a pluggable scorer, a concurrent runner with caching, and an aggregator that writes one deterministic `report.json` per run. None of that stops a regression from merging unless something actually reads the report and acts on it before the code ships — which, left to human diligence, happens reliably for about the first two weeks after the harness is built and then stops. This lesson wires the harness into CI so the decision is automatic: a pull request that regresses the agent's behavior does not merge, full stop, without anyone having to remember to check.

The obstacle is that CI's usual contract assumes determinism. A unit test either passes or it doesn't, and a failing test blocking a merge is uncontroversial because running it again produces the identical result. An agent eval case run through an LLM judge, or through an agent sampling at nonzero temperature, does not have that property — the same input can score differently on two runs with nothing in the code having changed. Bolt a nondeterministic suite onto a CI system built for deterministic tests and you get one of two failure modes: either the gate blocks merges on pure noise until engineers learn to ignore it, or someone quietly turns the gate off because it's ‘too flaky’, and you're back to the report nobody reads.

Getting this right is mostly a design problem, not a tooling problem: which suite runs at which point in the commit lifecycle, how a threshold absorbs real variance without absorbing real regressions, how the eval bill stays flat as commit volume grows, and what happens when the suite disagrees with itself on the same code twice. Get those decisions right and the gate becomes trustworthy enough that engineers stop re-running it hoping for green and start treating a failure as signal. Get them wrong and the gate becomes exactly the ignored report this lesson exists to prevent, just with more infrastructure underneath it.

Analogy🏏Cricket

🏏 Think of it like cricket: every net session at the NCA in Bengaluru produces a report — throwdown speeds, catches held, how many balls a batter middled out of fifty. For years those reports sat in a coach's notebook and nobody's selection changed because of them. Then the BCCI wired the numbers into an actual gate: a batter can't be picked for the India A tour unless his NCA fitness and skill numbers clear a fixed bar, checked automatically before the selection meeting even convenes. Yashasvi Jaiswal doesn't get a Ranji Trophy call-up because someone eyeballs his highlights reel; he gets it because his numbers cleared the bar the selectors pre-agreed on, every single time, without a selector having to personally re-derive the threshold from scratch. That's the difference between an eval report and an eval gate. A harness that prints a pass rate to a terminal after someone remembers to run it is the coach's notebook — useful, ignored, optional. The same harness wired to block a pull request the moment its numbers dip below an agreed threshold is the NCA's automatic cutoff — the check runs whether or not anyone thinks to look, and a regression can't quietly ride into the squad. Building the harness, as the last lesson did, is necessary. Making it a gate a PR cannot bypass is what actually stops a bad change from reaching production, the same way a cutoff a selector can quietly override protects nobody.