Most teams treat a golden eval set the way they treat a database migration: something you build once, ship, and mostly forget. That mental model breaks within a few months of real use, because everything an eval set depends on for its validity keeps moving underneath it — the model changes, the product it's testing changes, and the traffic that inspired the original cases changes. A 40-case eval set that gave an honest signal in month one can be actively misleading by month nine, and it fails silently: nobody gets paged when an eval set goes stale, the dashboard just keeps showing green.
This lesson covers the specific ways a golden set decays — contamination and leakage, saturation, and label rot — and the maintenance program that catches each one before it quietly stops telling you the truth. None of these are exotic failure modes; they're the predictable consequence of running the same 40 to 200 cases against a system that keeps changing for years. Lesson 3 covered building the first eval set; this lesson is about what happens to that set on its second birthday, and its fifth.
The fix is not more automation on top of a decaying set — it's discipline: per-case provenance metadata, a named owner, a review cadence, a dev/held-out split that's actually respected, and an honest accounting of what a case's removal does to the pass rate. Every technique below is cheap compared to the alternative, which is shipping a regression because the golden set that was supposed to catch it stopped measuring anything a year ago.
Analogy🏏Cricket
🏏 Think of it like cricket: A curator at Eden Gardens preparing a pitch for a five-day Test doesn't finish the job when the covers come off on the morning of day one. The surface that gave both Mohammed Shami's new-ball swing and Shubman Gill's cover drives a fair contest on day one is not the same surface by day four — footmarks form outside off stump, the top layer dries and starts to crumble, and by day five a part-time off-spinner can suddenly turn one square past the outside edge. A curator who treats 'the pitch is ready' as a one-time fact made on day one, then walks away, ends up with a wicket that either flattens into a shirtfront or crumbles into a minefield, and either way stops being a fair test of the two teams. The curator's job is a maintenance job: rolling, watering, and monitoring the surface through the match, not a preparation job that ends at the toss. A golden eval set behaves exactly like that pitch. The moment it ships, the model it tests keeps changing, the product behind it keeps changing, and the traffic feeding it keeps changing — so a set that was fair and discriminating on the day it launched quietly stops being either, unless someone keeps rolling it. The insight is that 'we built the eval set' is a day-one fact, not a permanent one, and a curator who stops watching the surface after the toss is the reason the Test stops being a fair contest by day four.
🏏 Showing the Cricket analogy — a Cricket version isn’t available for this concept yet.