A/B Testing and Shadow Deployment — MLOps & Model Deployment

Offline metrics tell you a new model is better on historical data, but they cannot tell you it is better for real users under real conditions, and deploying it to everyone at once on that faith is how good-looking models cause production disasters. A/B testing and shadow deployment are the techniques for validating a model in production safely, with real traffic, before it fully takes over. A/B testing routes a fraction of live traffic to the new model and compares outcomes against the current one on actual business metrics; shadow deployment sends a copy of live traffic to the new model without serving its predictions to users, so you observe its behaviour at zero user risk. The problem they solve is the chronic gap between offline promise and online reality: a model can win on a static test set yet fail live due to drift, latency, edge cases, or a metric that does not translate to user value. These techniques replace a risky all-at-once gamble with a controlled, evidence-based rollout, so promotion decisions rest on what the model actually does in production rather than on what a held-out dataset predicted it might do.

Analogy🏏Cricket

🏏 Think of it like cricket: imagine a batting coach analysing why Virat Kohli scored freely in one innings but struggled in another, yet kept no notes on which bat, which guard, or which net drills preceded each. Just as a detailed training diary logging bat weight, stance, and bowling type lets the coach link inputs to outputs, MLflow logs hyperparameters and data to outcomes. Just as comparing diary entries reveals that a heavier bat hurt timing, comparing tracked runs reveals which learning rate lifted accuracy. Just as a shared diary lets the whole support staff learn from one session, a shared tracking server lets the whole team learn from every run. The insight is that improvement is impossible without recorded cause and effect; tracking is what converts trial and error into knowledge.