Model Serving
Model serving is the process of deploying a trained machine learning model so it can receive input data and return predictions in a production environment, typically via an API.
Definition
Model serving is the process of deploying a trained machine learning model so it can receive input data and return predictions in a production environment, typically via an API.
Overview
Once a model has been trained and validated, it needs to be made available to applications that consume its predictions — this deployment and inference-serving layer is what model serving refers to. Model serving systems handle concerns that training pipelines don't: low-latency response times, high request throughput, concurrent request handling, versioning multiple models or model versions simultaneously, and efficient use of hardware accelerators like GPUs. Serving architectures generally fall into two categories: online (real-time) serving, where a client sends a request and receives a prediction within milliseconds to seconds, typically over a REST or gRPC API; and batch (offline) serving, where predictions are generated for a large set of inputs on a schedule and stored for later use, such as nightly recommendation score updates. Dedicated model-serving frameworks and platforms — such as TensorFlow Serving, TorchServe, NVIDIA Triton Inference Server, KServe (built on Kubernetes), and cloud-managed services like Amazon SageMaker, Google Vertex AI, and Azure Machine Learning — handle common infrastructure needs like autoscaling, request batching, model versioning, and canary rollouts, so teams don't have to build this tooling themselves. Modern model serving increasingly needs to support large language models specifically, which have unique serving challenges around GPU memory management, KV-cache handling, and request batching for variable-length generation; specialized inference engines like vLLM, TensorRT-LLM, and Text Generation Inference (TGI) address these needs with techniques like continuous batching and paged attention. Model serving is closely tied to model monitoring, since serving infrastructure is usually the layer where prediction logs, latency metrics, and input data are captured for downstream drift detection and quality tracking.
Key Concepts
- Deploys trained models to handle production inference requests
- Supports online (real-time) serving and batch (offline) serving patterns
- Handles concerns like latency, throughput, autoscaling, and concurrency
- Supports model versioning and canary or shadow rollouts of new versions
- Dedicated frameworks include TensorFlow Serving, TorchServe, and Triton
- Cloud-managed options include SageMaker, Vertex AI, and Azure ML
- LLM-specific serving engines (vLLM, TensorRT-LLM) optimize GPU memory and batching
- Provides the data capture point for downstream model monitoring
Use Cases
Frequently Asked Questions
From the Blog
The KV cache explained: why it dominates LLM memory during serving
The KV cache makes generation fast and memory-bound at once. Learn what it stores, why it caps concurrency, and the levers that shrink it without hurting quality.
Read More AI & TechnologyThe Main Types of Accounting Explained
Accounting splits into several distinct branches, including financial, managerial, tax, and auditing, each serving a different audience and purpose within an organization. This guide breaks down what each branch covers and how they relate.
Read More AI & TechnologyLLM Inference Optimization: Latency, Throughput and Cost
Latency, throughput and cost pull against each other in LLM serving, and most optimisation advice fails because it ignores which one you are actually optimising. This guide separates the three goals, maps each technique to the goal it moves, and names the quality or memory price each one charges.
Read More AI & TechnologyHow to Migrate a Vector Index Without Downtime
Migrate a vector index by writing to both the old and new index for a period, shadow-reading the new one to compare results, then cutting reads over behind a flag with the old index still warm. This keeps queries serving throughout and makes rollback a configuration change rather than a rebuild.
Read More