Inference
Inference is the process of running a trained machine learning model on new input data to produce a prediction, classification, or generated output. It is distinct from training, which is the earlier phase where the model learns its…
Definition
Inference is the process of running a trained machine learning model on new input data to produce a prediction, classification, or generated output. It is distinct from training, which is the earlier phase where the model learns its parameters from data; inference uses those already-learned, fixed parameters to compute a result in production.
Overview
Every machine learning system has two distinct phases: training and inference. Training is computationally intensive and happens once (or periodically, for retraining), during which a model iteratively adjusts millions or billions of parameters to minimize error on a training dataset. Inference happens every time the trained model is actually used — for example, each time a user sends a message to a chatbot or an image is classified — and applies the model's fixed, learned parameters to produce output for new input it has never seen before. For large language models, inference involves an autoregressive generation process: the model processes the input prompt and then generates output tokens one at a time, each new token predicted based on all preceding tokens (the prompt plus tokens generated so far). This sequential nature makes LLM inference latency-sensitive, and various techniques — such as key-value caching, batching, quantization, and specialized inference hardware or software (e.g., optimized inference servers) — are used to make it faster and cheaper at scale. Inference cost and speed are central engineering concerns in production AI systems. Larger models generally produce better quality output but cost more and are slower to run per request. This tradeoff drives decisions like model selection (choosing a smaller, cheaper model when acceptable), quantization (reducing numerical precision to speed up computation with minimal quality loss), and infrastructure choices (running inference on GPUs, TPUs, or specialized inference chips, on-premises or via cloud APIs). Understanding the training/inference distinction is fundamental: model quality is set once during training (or fine-tuning), while inference is what users actually interact with and what determines an application's real-time cost, latency, and responsiveness.
Key Concepts
- The process of applying a trained model's fixed parameters to new input to produce output
- Distinct from training, which is the earlier, far more compute-intensive learning phase
- For LLMs, generation is autoregressive: tokens are produced one at a time
- Latency and cost scale with model size, output length, and hardware used
- Optimized via techniques like caching, batching, and quantization
- Runs on specialized hardware: GPUs, TPUs, or dedicated inference accelerators
- Central to production AI system cost, speed, and scalability decisions
- Can be run in real time (online inference) or in bulk (batch inference)
Use Cases
Frequently Asked Questions
From the Blog
LLM Inference Optimization: Latency, Throughput and Cost
Latency, throughput and cost pull against each other in LLM serving, and most optimisation advice fails because it ignores which one you are actually optimising. This guide separates the three goals, maps each technique to the goal it moves, and names the quality or memory price each one charges.
Read More AI & TechnologyQuantising LLMs for inference: formats, trade-offs and how to validate
Quantisation trades precision for memory and speed. Learn weight-only versus activation quantisation, calibration, and how to prove quality has not degraded.
Read More AI & TechnologyWhat Is Model Quantization and Why It Matters
Model quantization shrinks a neural network by storing its weights in lower precision, cutting memory and speeding inference with little accuracy loss.
Read More AI & TechnologyHow Large Language Models Work, Explained Simply
How large language models work, explained simply: tokens, attention, training, and inference described with intuition and analogies, no heavy math required.
Read More