Drift: When Your Model or Data Changes Underneath You

An agent's eval score dropping in production rarely means someone broke the code. More often, the ground underneath the agent moved while the code stayed exactly the same: the model behind an alias got swapped, the documents in the retrieval index got re-chunked, users started asking a different mix of questions, or a colleague nudged the system prompt and shipped it without an eval run. Every one of these produces the identical symptom — a lower pass rate on last week's canary set — which is exactly why teams spend a debugging session guessing at code regressions that were never introduced.

Drift is not one phenomenon; it is at least six, and each kind needs a different fix. Input (covariate) drift means the questions changed but the correct answers didn't. Concept drift means the correct answer itself changed — a policy update, a new product, a pricing change — so an agent that used to be right is now confidently wrong. Model drift, tool drift, retrieval-corpus drift and prompt drift are all cases where a system you depend on, or your own system, changed underneath you without a deploy you initiated.

Conflating these wastes exactly the kind of debugging time this course has spent prior lessons teaching you to earn back with eval harnesses and observability traces. A team that sees a score drop and immediately starts re-reading the agent's own code is searching the one place that provably didn't change. This lesson gives you the vocabulary to name which of the six things moved, and the statistical and operational tooling — PSI, KS tests, embedding-space distance, canary eval sets, output-distribution monitoring and version pinning — to detect each one on a schedule instead of by user complaint.

Analogy🏏Cricket

🏏 Think of it like cricket: Imagine Jasprit Bumrah bowling the same length and seam position he always does at the death overs in a day-night ODI at the Wankhede. In the first innings, under the lights with a new white ball, his yorkers swing late and take wickets — figures of 4 for 22. He runs in exactly the same way in the second innings: same run-up, same wrist position, same release. But now the ball is old, the dew has crept onto the outfield, the seam has gone soft and won't bite. His stock delivery still swings, but it sits up nicely instead of jagging away, and the late order clubs him over midwicket for figures of 0 for 41 by the twentieth over. A commentator who only watches the economy rate says 'Bumrah has lost his mojo' — the same mistake a team makes when an agent's eval score drops and everyone assumes the prompt or the code regressed. Nobody touched Bumrah's action. The conditions moved underneath the same delivery. That's drift: the model, the users, the world, or your own systems change quietly, and the thing you shipped is no longer the thing that is running.