Serving Practice: Model API with Autoscaling

What You'll Build

In this exercise you will build a production-style model serving API for the cricket in-form classifier, containerise it, and configure it to autoscale under load. You will wrap an ONNX model in a FastAPI service with a validated request contract, health and readiness endpoints, and basic metrics, package it into a lean Docker image, then define the autoscaling policy that adds and removes replicas as request volume changes. This pulls together packaging, serving, and the operational concerns of running a model under real traffic into one deployable unit. By the end you will have an endpoint that validates inputs, loads the model once per worker, reports whether it is healthy, and scales horizontally so it stays fast during traffic spikes without paying for idle capacity during quiet periods, the shape of a real production inference service rather than a notebook demo.

Analogy🏏Cricket

🏏 Think of it like cricket: a selection committee does not pick a squad on a hunch; they run structured trial matches under recorded conditions, log every player's scores, compare candidates on identical criteria, and keep the records so a selection can be justified later. Just as the trial matches are your tracked experiment runs, each player's logged scores are your run metrics. Just as the committee compares candidates on the same pitch to be fair, you compare model configurations on the same data. Just as a defensible selection can be reproduced from the records if challenged, your pipeline can reproduce any run from its logged inputs. The insight is that disciplined, recorded trials, not gut feel, are what make both squad selection and model selection defensible.

Prerequisites

Completion of lessons 11 and 13 (packaging and REST serving), or equivalent familiarity with Docker and FastAPI.
Python 3.11, Docker installed and running, and the ability to build and run a local container.
An ONNX model file (form_net.onnx) such as the one exported in lesson 11, or any compatible classifier.
Basic understanding of HTTP requests, JSON, and what horizontal scaling and replicas mean.
Optional: access to a Kubernetes cluster (or a local one like kind or minikube) to apply the autoscaling manifest.

Setup & Project Structure

You will create a serving project that cleanly separates the application code, the container definition, and the deployment manifests, the layout a real serving repository uses. The FastAPI app and its model live together, the Dockerfile packages them, and Kubernetes manifests describe the deployment and its autoscaling policy. Keeping the deployment configuration separate from the application matters because the same image is promoted unchanged across environments while only the scaling and resource settings differ, exactly the separation that lets one validated artifact run consistently from local testing to production. Lay out the folders and gather the dependencies before writing the service.

Analogy🏏Cricket

🏏 Think of it like cricket: a well-run academy never dumps kit, players, and match records into one heap; it keeps the practice ground, the coaching staff's playbook, and the scorers' logbook in separate, clearly labelled areas so anyone can find what they need and nothing gets mixed up. Just as you separate the data directory, the src pipeline logic, and the mlruns tracking store, the academy separates its pitches, its coaching manuals, and its performance ledgers. Just as keeping these concerns apart makes a project's structure obvious at a glance, a tidy academy lets a new coach walk in and immediately know where drills, plans, and records live. Just as MLflow writes runs to a local store beside, not inside, the code, the scorers keep the logbook outside the coaching manual so results never overwrite strategy. The payoff: a clean layout means a stranger, or you in six months, can pick up the project and understand exactly how data becomes a model.

bash

# Create the serving project skeleton.
mkdir -p cricket-serving/{app,k8s}
cd cricket-serving
cp /path/to/form_net.onnx app/        # the model exported in lesson 11

# requirements.txt (pinned for reproducibility)
cat > app/requirements.txt <<'EOF'
fastapi==0.111.0
uvicorn[standard]==0.30.0
onnxruntime==1.18.0
numpy==1.26.4
prometheus-client==0.20.0
EOF

# Resulting structure:
# cricket-serving/
# |-- app/
# |   |-- serve.py           # Step 1-2: FastAPI service + metrics
# |   |-- form_net.onnx      # the model artifact
# |   `-- requirements.txt
# |-- Dockerfile             # Step 3: lean, layer-cached image
# `-- k8s/
#     |-- deployment.yaml    # Step 3: deployment + resource requests
#     `-- hpa.yaml           # Step 3: horizontal pod autoscaler

echo 'Serving project skeleton ready.'

Step 1 — Foundation

Step 1 builds the foundation of the service: loading the model once at startup and defining the validated request and response contract. The concept behind this step is that a serving endpoint's correctness and performance both start here, the model must be a long-lived resource loaded a single time per worker, and every incoming request must be validated against a typed schema before it ever reaches inference. Getting this foundation right prevents the two most common serving failures at once: the per-request model reload that collapses under load, and the malformed payload that produces garbage predictions. This step establishes the safe, efficient base that everything else builds on.

Analogy🏏Cricket

🏏 Think of it like cricket: before any trial match, the groundsman prepares one agreed pitch and the same set of balls, so every candidate is judged on identical conditions rather than a surface that changes underfoot. Just as the fixed pitch is your versioned dataset, the standard ball is your deterministic feature function. Just as changing the pitch mid-trial would make scores incomparable, a feature function with hidden randomness would make runs incomparable. Just as recording the pitch and ball used lets you recreate the conditions, recording the data version lets you recreate the inputs. The insight is that fair, repeatable comparison demands a fixed, documented starting surface.

python

# app/serve.py (part 1)  -- Step 1: load once + validated contract.
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
import numpy as np
import onnxruntime as ort

app = FastAPI(title='Cricket Form Serving')

# Loaded ONCE at import (per worker process), reused across all requests.
_session = ort.InferenceSession('form_net.onnx', providers=['CPUExecutionProvider'])
_ready = True   # flips false if the model fails to load / a dependency is down

class InningsFeatures(BaseModel):
    batting_average: float = Field(..., ge=0, le=120)
    strike_rate: float = Field(..., ge=0, le=400)
    boundary_pct: float = Field(..., ge=0, le=1)

class FormPrediction(BaseModel):
    in_form: bool
    confidence: float

def _infer(features: InningsFeatures) -> FormPrediction:
    x = np.array([[features.batting_average, features.strike_rate,
                   features.boundary_pct]], dtype=np.float32)
    logits = _session.run(['logits'], {'features': x})[0][0]
    probs = np.exp(logits) / np.exp(logits).sum()
    cls = int(np.argmax(probs))
    return FormPrediction(in_form=bool(cls), confidence=float(probs[cls]))

Step 2 — Core Logic

Step 2 adds the serving endpoints and observability: the prediction route, health and readiness probes, and request metrics. This is the core logic that makes the service operable under an orchestrator and an autoscaler, the prediction endpoint delivers the actual value, the liveness and readiness endpoints let the platform know whether to send traffic, and the metrics expose request count and latency so the autoscaler and operators can see load. Without health probes an orchestrator cannot tell a starting or broken pod from a healthy one, and without metrics there is nothing for an autoscaler to react to. This step turns the foundation into a fully operable endpoint.

Analogy🏏Cricket

🏏 Think of it like cricket: when a candidate bats in a trial, an official scorer records the conditions they batted in, the pitch, the bowling, and their exact score, all under one innings entry. Just as the recorded conditions are your logged params, the score is your logged metric, and the innings entry is the MLflow run. Just as the same conditions and the same player should reproduce a comparable knock, the same params and seed reproduce the same accuracy. Just as a scorer who noted only the score but not the conditions leaves selectors guessing, logging a metric without its params leaves you unable to explain a result. The insight is that a result is only useful when recorded together with the conditions that produced it.

python

# app/serve.py (part 2)  -- Step 2: endpoints, probes, metrics.
from fastapi import Response
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
import time

REQUESTS = Counter('predict_requests_total', 'Total prediction requests')
LATENCY = Histogram('predict_latency_seconds', 'Prediction latency')

@app.post('/predict', response_model=FormPrediction)
async def predict(features: InningsFeatures) -> FormPrediction:
    REQUESTS.inc()
    start = time.perf_counter()
    try:
        return _infer(features)
    except Exception as e:
        raise HTTPException(status_code=500, detail=f'inference failed: {e}')
    finally:
        LATENCY.observe(time.perf_counter() - start)

@app.get('/healthz')          # liveness: is the process up?
async def healthz():
    return {'status': 'alive'}

@app.get('/readyz')           # readiness: should we receive traffic?
async def readyz():
    if not _ready:
        raise HTTPException(status_code=503, detail='model not ready')
    return {'status': 'ready'}

@app.get('/metrics')          # scraped by Prometheus -> drives autoscaling
async def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

# Run locally:  uvicorn serve:app --host 0.0.0.0 --port 8000 --workers 2

Step 3 — Integration & Enhancement

Step 3 brings it together: package the service into a lean Docker image and define the Kubernetes deployment with resource requests and a horizontal pod autoscaler. This integration is what makes the endpoint production-ready, the image is the portable artifact, the deployment declares how many replicas and how much CPU each needs, and the autoscaler ties replica count to observed load so the service grows under traffic and shrinks when quiet. Setting CPU requests correctly is essential because the autoscaler scales on CPU utilisation relative to those requests; without sensible requests the autoscaler cannot reason about load. This step turns the operable endpoint into an elastic, self-scaling service.

Analogy🏏Cricket

🏏 Think of it like cricket: after a round of trial matches, the committee lays every candidate's recorded scores side by side, picks the top performer on the agreed metric, and formally names them to the squad while keeping the rest on record. Just as laying out all scorecards is your sweep across configs, choosing the highest scorer is your selection by logged accuracy. Just as the named player is promoted while others remain in the pool for recall, the best model is promoted while others stay tracked. Just as the selection memo cites the scores that justified it, your promotion cites the winning run ID. The insight is that evidence-based selection across recorded candidates produces a defensible, reversible choice.

python

# Dockerfile  -- Step 3a: lean, layer-cached image.
dockerfile = '''
FROM python:3.11-slim
WORKDIR /app
COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # cached unless deps change
COPY app/ .                                          # code+model layered last
EXPOSE 8000
CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "2"]
'''

# k8s/deployment.yaml  -- Step 3b: deployment with probes + resource requests.
deployment_yaml = '''
apiVersion: apps/v1
kind: Deployment
metadata: { name: cricket-form }
spec:
  replicas: 2
  selector: { matchLabels: { app: cricket-form } }
  template:
    metadata: { labels: { app: cricket-form } }
    spec:
      containers:
      - name: server
        image: myregistry.io/cricket-form:1.0.0
        ports: [ { containerPort: 8000 } ]
        resources:
          requests: { cpu: "250m", memory: "256Mi" }   # autoscaler baseline
          limits:   { cpu: "1",    memory: "512Mi" }
        readinessProbe: { httpGet: { path: /readyz, port: 8000 }, initialDelaySeconds: 5 }
        livenessProbe:  { httpGet: { path: /healthz, port: 8000 }, initialDelaySeconds: 10 }
'''

# k8s/hpa.yaml  -- Step 3c: autoscale on CPU utilisation.
hpa_yaml = '''
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: cricket-form-hpa }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: cricket-form }
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 70 }   # add pods past 70%
'''
print('Image + deployment + HPA: an elastic, self-scaling serving service.')

Step 4 — Testing & Verification

Verify three things: the container builds and serves correct predictions, the health and readiness endpoints respond appropriately, and the autoscaler adds replicas under sustained load. Build and run the image locally, send a valid and an invalid request to confirm validation works, hit the probes, then deploy to a cluster and generate load while watching the replica count climb. This confirms the whole chain, from packaged model to validated endpoint to elastic scaling, is working end to end.

Analogy🏏Cricket

🏏 Think of it like cricket: before trusting a match plan you verify two things, that it actually works start to finish under real conditions and produces a clear winner, and that it is repeatable, run the same trial again in the same conditions and you get the same result rather than a fluke. Just as you confirm the pipeline runs end to end and produces a tracked leaderboard plus a promoted model, a coach confirms the full session runs from warm-up to selection and yields a ranked shortlist with a clear pick. Just as you rerun the same configuration and check it yields the same accuracy, a coach reruns the identical trial and expects the same player to top the table, proving it was skill, not chance. Just as the MLflow UI records every run with its params, metric, and artifact, the scorebook records every trial so the result can be inspected and defended. The payoff: verifying both correctness and reproducibility, in a pipeline or a trial, means you trust the winner because it holds up when repeated.

bash

# Build, run, and exercise the service; then watch it autoscale.
cd cricket-serving
docker build -t cricket-form:1.0.0 .
docker run -d -p 8000:8000 cricket-form:1.0.0

# Valid request -> 200 with a prediction:
curl -s -X POST localhost:8000/predict -H 'content-type: application/json' \
  -d '{"batting_average": 53.6, "strike_rate": 131.0, "boundary_pct": 0.61}'
# Expected: {"in_form": true, "confidence": 0.9x}

# Invalid request (strike_rate out of range) -> 422 from schema validation:
curl -s -X POST localhost:8000/predict -H 'content-type: application/json' \
  -d '{"batting_average": 53.6, "strike_rate": 999, "boundary_pct": 0.61}'
# Expected: 422 Unprocessable Entity

curl -s localhost:8000/readyz     # -> {"status":"ready"}
curl -s localhost:8000/healthz    # -> {"status":"alive"}

# On a cluster: deploy and watch the autoscaler react to load.
kubectl apply -f k8s/deployment.yaml -f k8s/hpa.yaml
kubectl get hpa cricket-form-hpa --watch     # REPLICAS rises as load grows
# Generate load (e.g. with hey):  hey -z 60s -c 50 -m POST -D body.json \
#   http://<service>/predict
# Expected: replica count climbs from 2 toward 10 while CPU > 70%, then settles back.

Warning: A common error is omitting CPU resource requests on the deployment, then wondering why the horizontal pod autoscaler never scales. The HPA computes utilisation as actual CPU divided by the requested CPU, so with no request there is no denominator and the autoscaler has no signal to act on. Always set realistic resource requests; they are what define what 'busy' means per replica and are the basis of every scaling decision.

Extension Challenge: Replace CPU-based scaling with a custom metric, scale on predict_requests_total per second via the Prometheus Adapter, so the service reacts directly to request rate rather than CPU. For a harder stretch, add a small dynamic batching layer that groups requests arriving within a 15ms window into one ONNX call, then compare throughput and tail latency with and without batching under the same load test.

A production serving endpoint loads the model once per worker, validates inputs against a typed schema, and exposes health, readiness, and metrics.
Liveness and readiness probes let the orchestrator restart broken pods and route traffic only to ready ones.
Containerising with a lean, layer-cached image produces a portable artifact promoted unchanged across environments.
A horizontal pod autoscaler adds and removes replicas based on observed load, keeping latency low under spikes without paying for idle peak capacity.
CPU resource requests define per-replica utilisation and are the denominator the autoscaler uses, so the HPA cannot scale without them.
Keeping deployment and scaling config separate from application code lets one validated image run consistently from local testing to production.