Model Serving
Model serving is the process of deploying a trained machine learning model so it can receive input data and return predictions in a production environment, typically via an API.
Definition
Model serving is the process of deploying a trained machine learning model so it can receive input data and return predictions in a production environment, typically via an API.
Overview
Once a model has been trained and validated, it needs to be made available to applications that consume its predictions — this deployment and inference-serving layer is what model serving refers to. Model serving systems handle concerns that training pipelines don't: low-latency response times, high request throughput, concurrent request handling, versioning multiple models or model versions simultaneously, and efficient use of hardware accelerators like GPUs. Serving architectures generally fall into two categories: online (real-time) serving, where a client sends a request and receives a prediction within milliseconds to seconds, typically over a REST or gRPC API; and batch (offline) serving, where predictions are generated for a large set of inputs on a schedule and stored for later use, such as nightly recommendation score updates. Dedicated model-serving frameworks and platforms — such as TensorFlow Serving, TorchServe, NVIDIA Triton Inference Server, KServe (built on Kubernetes), and cloud-managed services like Amazon SageMaker, Google Vertex AI, and Azure Machine Learning — handle common infrastructure needs like autoscaling, request batching, model versioning, and canary rollouts, so teams don't have to build this tooling themselves. Modern model serving increasingly needs to support large language models specifically, which have unique serving challenges around GPU memory management, KV-cache handling, and request batching for variable-length generation; specialized inference engines like vLLM, TensorRT-LLM, and Text Generation Inference (TGI) address these needs with techniques like continuous batching and paged attention. Model serving is closely tied to model monitoring, since serving infrastructure is usually the layer where prediction logs, latency metrics, and input data are captured for downstream drift detection and quality tracking.
Key Concepts
- Deploys trained models to handle production inference requests
- Supports online (real-time) serving and batch (offline) serving patterns
- Handles concerns like latency, throughput, autoscaling, and concurrency
- Supports model versioning and canary or shadow rollouts of new versions
- Dedicated frameworks include TensorFlow Serving, TorchServe, and Triton
- Cloud-managed options include SageMaker, Vertex AI, and Azure ML
- LLM-specific serving engines (vLLM, TensorRT-LLM) optimize GPU memory and batching
- Provides the data capture point for downstream model monitoring