Model Deployment Basics Cheat Sheet
Explains how to serve trained models via REST APIs, containerize them with Docker, and choose between batch, real-time, and canary deployment patterns.
2 PagesIntermediateMar 2, 2026
Serving a Model with FastAPI
Wrap a trained model in a REST API endpoint.
python
from fastapi import FastAPIfrom pydantic import BaseModelimport joblibimport numpy as npapp = FastAPI()model = joblib.load("model.pkl")class PredictRequest(BaseModel): features: list[float]@app.post("/predict")def predict(req: PredictRequest): X = np.array(req.features).reshape(1, -1) pred = model.predict(X) return {"prediction": pred.tolist()}# Run with: uvicorn app:app --host 0.0.0.0 --port 8000
Containerizing the Service
Package the model API into a portable Docker image.
dockerfile
FROM python:3.11-slimWORKDIR /appCOPY requirements.txt .RUN pip install --no-cache-dir -r requirements.txtCOPY model.pkl app.py ./EXPOSE 8000CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
Deployment Patterns
Common strategies for rolling out a model safely.
- Batch inference- Run predictions on a schedule (e.g. nightly) over a dataset and write results to storage
- Online/real-time inference- Model served behind an API endpoint for low-latency, per-request predictions
- Blue-green deployment- Run old (blue) and new (green) versions side by side, switch traffic once green is verified
- Canary release- Route a small percentage of traffic to the new model version before a full rollout
- Shadow deployment- New model runs alongside the production model on live traffic without affecting responses, for comparison
- Model registry- Central store (e.g. MLflow Model Registry) that versions models and tracks stage (staging/production)
Monitoring & Versioning
What to track once a model is live.
- Data drift- Statistical change in input feature distributions compared to training data
- Concept drift- Change in the relationship between inputs and the target over time, degrading model accuracy
- Latency/throughput monitoring- Track p50/p95/p99 response times and requests per second for the serving endpoint
- Model versioning- Tag each deployed artifact with a version/hash so predictions are reproducible and rollback-able
- A/B testing- Compare business metrics between model versions on live, randomly split traffic
Multi-Model Serving with Triton Inference Server
Configure dynamic batching and concurrent model instances for high-throughput GPU serving.
yaml
name: "fraud_detector"platform: "onnxruntime_onnx"max_batch_size: 64dynamic_batching { preferred_batch_size: [8, 16, 32] max_queue_delay_microseconds: 2000}instance_group [ { count: 2 kind: KIND_GPU gpus: [0] }]optimization { execution_accelerators { gpu_execution_accelerator: [ { name: "tensorrt" parameters { key: "precision_mode" value: "FP16" } } ] }}# Launch: tritonserver --model-repository=/models --log-verbose=1
Progressive Canary Rollout with KServe
Split live inference traffic between two model revisions using a Kubernetes InferenceService.
yaml
apiVersion: serving.kserve.io/v1beta1kind: InferenceServicemetadata: name: churn-modelspec: predictor: canaryTrafficPercent: 10 model: modelFormat: name: sklearn storageUri: "s3://models/churn/v2" resources: requests: cpu: "500m" memory: "512Mi" limits: cpu: "1" memory: "1Gi"---# Promote once metrics look good:# kubectl patch isvc churn-model --type merge \# -p '{"spec":{"predictor":{"canaryTrafficPercent":0}}}'
Exporting and Quantizing a Model for Lightweight Serving
Convert a PyTorch model to ONNX and apply dynamic INT8 quantization to cut inference latency.
python
import torchfrom onnxruntime.quantization import quantize_dynamic, QuantTypedummy_input = torch.randn(1, 20)torch.onnx.export( model, dummy_input, "model.onnx", input_names=["features"], output_names=["logits"], dynamic_axes={"features": {0: "batch"}, "logits": {0: "batch"}}, opset_version=17,)quantize_dynamic( model_input="model.onnx", model_output="model_int8.onnx", weight_type=QuantType.QInt8,)# INT8 model is typically 3-4x smaller and 2-4x faster on CPU
Avoiding Cold Starts with Warm-Pool Preloading
Load model weights at container startup and expose a readiness probe so orchestrators never route traffic to an unready pod.
python
from fastapi import FastAPIimport joblib, threadingapp = FastAPI()state = {"ready": False, "model": None}def warm_load(): state["model"] = joblib.load("model.pkl") _ = state["model"].predict([[0.0] * 20]) # trigger JIT/lazy init paths state["ready"] = True@app.on_event("startup")def startup(): threading.Thread(target=warm_load, daemon=True).start()@app.get("/healthz/ready")def readiness(): return {"ready": state["ready"]}, (200 if state["ready"] else 503)# Kubernetes readinessProbe hits /healthz/ready before adding the pod to the Service endpoints
Serving Runtime Trade-offs
Choosing an inference stack beyond a plain FastAPI wrapper.
- Dynamic batching- Server groups concurrent requests into micro-batches to raise GPU utilization at the cost of a few milliseconds of queuing latency
- gRPC vs REST- gRPC's binary protobuf framing and HTTP/2 multiplexing cuts serialization overhead vs JSON/REST for high-QPS or large-tensor payloads
- Multi-model endpoints- One serving process hosts many small models and lazy-loads/evicts them (LRU) to control memory footprint on shared infrastructure
- Autoscaling on queue depth- Scale replicas on inference queue length or p99 latency rather than raw CPU%, since GPU-bound workloads saturate CPU poorly as a signal
- Model warm pools- Keep a minimum replica count always loaded to eliminate cold-start latency spikes after scale-to-zero events
- Feature skew guardrails- Validate incoming request schemas/ranges against the training feature statistics at request time and reject or flag out-of-distribution inputs
- Rollback contract- Every deployment must be reversible to the prior registry version with a single command/API call, tested before go-live, not after an incident
Pro Tip
Always log the model version and input feature values alongside each prediction - without this, you cannot reproduce or debug a bad prediction after the fact.
Was this cheat sheet helpful?
Explore Topics
#ModelDeploymentBasics#ModelDeploymentBasicsCheatSheet#DataScience#Intermediate#ServingAModelWithFastAPI#ContainerizingTheService#DeploymentPatterns#MonitoringVersioning#MachineLearning#APIs#Docker#DevOps#CheatSheet#SkillVeris
