Batch Inference vs Online Inference — MLOps & Model Deployment

Not every prediction needs to happen the instant a user clicks; some can be computed in bulk overnight, while others must return in milliseconds, and choosing between batch and online inference is one of the most consequential architectural decisions in deploying a model. Batch inference scores large sets of inputs periodically, writing results to a store for later lookup, whereas online inference scores one input on demand behind a low-latency API. The problem this distinction addresses is matching the serving pattern to the use case: serving recommendations precomputed nightly is wasteful to recompute per request, while fraud detection on a live transaction cannot wait for a nightly job. Getting this choice wrong is expensive in both directions, building a complex real-time stack for predictions that could be precomputed, or trying to serve genuinely real-time needs from stale batch tables. Understanding the trade-offs in latency, cost, freshness, and complexity is what lets you design a serving architecture that meets the actual requirement efficiently rather than over- or under-engineering it.

Analogy🏏Cricket

🏏 Think of it like cricket: imagine a batting coach analysing why Virat Kohli scored freely in one innings but struggled in another, yet kept no notes on which bat, which guard, or which net drills preceded each. Just as a detailed training diary logging bat weight, stance, and bowling type lets the coach link inputs to outputs, MLflow logs hyperparameters and data to outcomes. Just as comparing diary entries reveals that a heavier bat hurt timing, comparing tracked runs reveals which learning rate lifted accuracy. Just as a shared diary lets the whole support staff learn from one session, a shared tracking server lets the whole team learn from every run. The insight is that improvement is impossible without recorded cause and effect; tracking is what converts trial and error into knowledge.