Inference
Inference is the process of running a trained machine learning model on new input data to produce a prediction, classification, or generated output. It is distinct from training, which is the earlier phase where the model learns its…
Definition
Inference is the process of running a trained machine learning model on new input data to produce a prediction, classification, or generated output. It is distinct from training, which is the earlier phase where the model learns its parameters from data; inference uses those already-learned, fixed parameters to compute a result in production.
Overview
Every machine learning system has two distinct phases: training and inference. Training is computationally intensive and happens once (or periodically, for retraining), during which a model iteratively adjusts millions or billions of parameters to minimize error on a training dataset. Inference happens every time the trained model is actually used — for example, each time a user sends a message to a chatbot or an image is classified — and applies the model's fixed, learned parameters to produce output for new input it has never seen before. For large language models, inference involves an autoregressive generation process: the model processes the input prompt and then generates output tokens one at a time, each new token predicted based on all preceding tokens (the prompt plus tokens generated so far). This sequential nature makes LLM inference latency-sensitive, and various techniques — such as key-value caching, batching, quantization, and specialized inference hardware or software (e.g., optimized inference servers) — are used to make it faster and cheaper at scale. Inference cost and speed are central engineering concerns in production AI systems. Larger models generally produce better quality output but cost more and are slower to run per request. This tradeoff drives decisions like model selection (choosing a smaller, cheaper model when acceptable), quantization (reducing numerical precision to speed up computation with minimal quality loss), and infrastructure choices (running inference on GPUs, TPUs, or specialized inference chips, on-premises or via cloud APIs). Understanding the training/inference distinction is fundamental: model quality is set once during training (or fine-tuning), while inference is what users actually interact with and what determines an application's real-time cost, latency, and responsiveness.
Key Concepts
- The process of applying a trained model's fixed parameters to new input to produce output
- Distinct from training, which is the earlier, far more compute-intensive learning phase
- For LLMs, generation is autoregressive: tokens are produced one at a time
- Latency and cost scale with model size, output length, and hardware used
- Optimized via techniques like caching, batching, and quantization
- Runs on specialized hardware: GPUs, TPUs, or dedicated inference accelerators
- Central to production AI system cost, speed, and scalability decisions
- Can be run in real time (online inference) or in bulk (batch inference)