100% Free Forever
AI-Powered Learning
Industry Expert Content
Certificates & Badges
Learn At Your Own Pace
MLOps & Model Deployment
60 minadvanced

Mid-Course Project: End-to-End MLOps Pipeline

In this mid-course project you will build an end-to-end MLOps pipeline for a cricket in-form prediction service, assembling the lifecycle from Modules 1 through 4 into one coherent system. The pipeline versions its data, tracks every training run, registers and promotes the best model, serves it behind a validated API, and monitors the live predictions for drift, feeding a signal back to trigger retraining. Rather than demonstrating each piece in isolation, you will connect them so an artifact flows from raw data all the way to a monitored production endpoint with full lineage. This is the portfolio centrepiece of the first half of the course: it proves you can not only use individual MLOps tools but architect them into a working whole, the single most valuable and most commonly tested skill for an MLOps engineer. By the end you will have a repository that takes raw cricket statistics and produces a reproducible, tracked, served, and monitored model, the concrete shape of what a production ML team actually ships and operates day to day.

Analogy🏏Cricket
🏏 Think of it like cricket: running an international team is not about excelling at any single function, scouting, selection, coaching, match tactics, or post-match review, but about wiring them into one continuous operation where each feeds the next across a whole season. Just as a great cricket operation connects scouting into selection into coaching into match play into review and back into scouting, your pipeline connects data versioning into tracking into registry into serving into monitoring and back into retraining. Just as a team that does each function brilliantly but never links them wins nothing, MLOps tools used in isolation deliver no production system. Just as the season's loop is what sustains performance, the pipeline's loop is what sustains the model. The insight is that mastery is demonstrated by integration, making the parts work together as a living system, not by polishing any part alone.

Learning Objectives

  • Architect a complete MLOps pipeline that connects data versioning, experiment tracking, model registry, serving, and monitoring into one flow.
  • Produce reproducible training runs whose data version, parameters, metrics, and resulting model are fully tracked and linked by lineage.
  • Promote models through registry stages and serve the production version by stage reference rather than a hard-coded artifact.
  • Expose the model behind a validated REST API with health and readiness endpoints and prediction logging for monitoring.
  • Detect data drift on the live prediction stream and emit a signal that triggers retraining, closing the lifecycle loop.
  • Reason about and demonstrate the handoffs between components, where lineage, freshness, and safety must be preserved across boundaries.

Technical Requirements

  • A versioned cricket dataset of player innings features (batting_average, strike_rate, boundary_pct) with an in-form label, tracked so each run references a specific version.
  • An experiment-tracking layer (MLflow) logging params, the data version, the accuracy metric, and the model artifact for every training run.
  • A model registry with at least a Staging and Production stage, where promotion compares the candidate against the incumbent on a recent evaluation set.
  • A FastAPI serving endpoint that loads the Production model by stage reference, validates inputs with a typed schema, and exposes /predict, /healthz, and /readyz.
  • Prediction logging that records each request's features and output to a store the monitor can read.
  • A drift monitor that compares the live feature window against the training reference and returns a pass/fail drift verdict.
  • A retraining trigger that fires on detected drift or accumulated new labelled data, runs the training pipeline, and routes a better candidate to Staging.
  • Reproducibility throughout: a single seed and a recorded data version make any run and any served model recoverable.

Architecture & Design

The pipeline is organised as five components connected by clear, durable interfaces, so each can evolve independently while the system stays coherent. The data layer owns a versioned dataset and a deterministic feature function, exposing a labelled, reproducible training set keyed by a data version. The training-and-tracking component consumes a data version, trains a model, and logs the run, params, the data version, accuracy, and the model artifact, to the tracker, emitting a run ID. The registry component takes a run's model, compares it against the current Production model on a recent evaluation slice, and on success promotes it through Staging to Production, holding lineage back to the run. The serving component loads the Production model strictly by stage reference, validates each request against a typed schema, returns a prediction, and logs the request features and output. The monitoring component reads that prediction log, compares the live feature distribution against the training reference, and emits a drift verdict that the retraining trigger consumes to decide whether to launch a new training run, closing the loop. The critical design principle is that components communicate through stable artifacts and references, a data version, a run ID, a registry stage, a prediction log, never by reaching into each other's internals, so lineage, freshness, and safety are preserved at every boundary and any component can be swapped without rewiring the rest.

Analogy🏏Cricket
🏏 Think of it like cricket: a national setup runs as distinct departments, scouting, selection, coaching, match management, analytics, that interact only through defined handoffs: scouting files reports, selectors issue a squad, coaches receive that squad, match management fields the eleven, analysts file a review. Just as each department hands the next a defined artifact (a report, a squad sheet, a review) rather than meddling in its work, each pipeline component hands over a defined artifact (a data version, a run ID, a registry stage). Just as a clear handoff lets the coaching staff change methods without disrupting selection, stable interfaces let serving change without disrupting training. Just as the review feeds back into the next scouting cycle, monitoring feeds back into retraining. The insight is that a robust operation is built from well-bounded units joined by clean handoffs, which is exactly how the pipeline's components connect.
python
# Architecture skeleton: five components joined by stable interfaces.
# Project structure:
# cricket-mlops/
# |-- data/            versioned dataset + deterministic features
# |-- pipeline/
# |   |-- data.py          (DataLayer)      -> labelled set keyed by data_version
# |   |-- train.py         (Trainer)        -> trains + logs run, returns run_id
# |   |-- registry.py      (Registry)       -> compare + promote by stage
# |   |-- monitor.py       (DriftMonitor)   -> live vs reference -> drift verdict
# |   `-- trigger.py       (RetrainTrigger) -> drift/volume -> launch training
# |-- serving/
# |   `-- serve.py         (FastAPI)        -> load Production by stage + log preds
# `-- run_pipeline.py      orchestrates the loop

from dataclasses import dataclass
from typing import Protocol

# Interfaces (the handoffs between components):
class DataLayer(Protocol):
    def training_set(self, data_version: str): ...      # -> (X, y)
    def reference_features(self): ...                   # -> baseline for drift

class Trainer(Protocol):
    def train_and_track(self, data_version: str) -> str: ...   # -> run_id

class Registry(Protocol):
    def promote_if_better(self, run_id: str, eval_X, eval_y) -> str: ...  # -> stage
    def load_production(self): ...                      # by stage, not by file

class DriftMonitor(Protocol):
    def drift_detected(self, live_features) -> bool: ...

@dataclass
class Pipeline:
    data: DataLayer
    trainer: Trainer
    registry: Registry
    monitor: DriftMonitor
    # run_pipeline.py wires these together; each talks only via the interfaces above.

Phase 1 — Core Implementation

Phase 1 implements the offline backbone: the data layer, the tracked trainer, and the registry with a comparison-gated promotion. This is the core because it establishes the lineage spine, a versioned dataset feeding tracked runs feeding a registry, on which serving and monitoring later hang. You will build a deterministic feature function over a versioned dataset, a trainer that logs the data version, params, accuracy, and model to MLflow and returns a run ID, and a registry that promotes a run's model only if it beats the incumbent on an evaluation slice. Getting this phase right means any model can be traced back to the exact data and run that produced it, the non-negotiable foundation of the whole pipeline.

Analogy🏏Cricket
🏏 Think of it like cricket: before a season you establish the backbone of the operation, a fixed player database, a recorded selection process, and a squad list that traces every pick back to the trials that justified it. Just as the player database is your versioned data and the recorded selection is your tracked training, the traceable squad list is your registry. Just as a squad with no record of why each player was picked cannot be defended or revised, a model with no lineage cannot be audited or rolled back. Just as this backbone underpins every match decision that follows, this offline spine underpins serving and monitoring. The insight is that establishing a traceable selection backbone first is what makes everything downstream accountable and revisable.
python
# Phase 1: data layer + tracked trainer + comparison-gated registry promotion.
import mlflow, mlflow.sklearn
import random
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

SEED = 1983
MODEL_NAME = 'PlayerFormClassifier'
mlflow.set_experiment('cricket-mlops-midcourse')

class DataLayer:
    def __init__(self, data_version='[email protected]'):
        self.data_version = data_version
        rng = random.Random(SEED)
        self._rows = [([rng.uniform(10, 65), rng.uniform(60, 160), rng.uniform(0.1, 0.75)],
                       0) for _ in range(400)]
        for r in self._rows:                       # deterministic label rule
            avg, sr, bp = r[0]
            r_label = 1 if (0.03*avg + 0.02*sr + 1.5*bp) > 4.0 else 0
            r_index = self._rows.index(r)
            self._rows[r_index] = (r[0], r_label)
    def training_set(self):
        return [r[0] for r in self._rows], [r[1] for r in self._rows]
    def reference_features(self):
        return [r[0][0] for r in self._rows]        # batting_average baseline

class Trainer:
    def __init__(self, data: DataLayer):
        self.data = data
    def train_and_track(self, max_depth=4) -> str:
        X, y = self.data.training_set()
        with mlflow.start_run() as run:
            mlflow.log_param('data_version', self.data.data_version)   # lineage
            mlflow.log_param('max_depth', max_depth)
            model = DecisionTreeClassifier(max_depth=max_depth, random_state=SEED).fit(X, y)
            mlflow.log_metric('accuracy', accuracy_score(y, model.predict(X)))
            mlflow.sklearn.log_model(model, 'model')
            return run.info.run_id

class Registry:
    def promote_if_better(self, run_id, eval_X, eval_y):
        candidate = mlflow.sklearn.load_model(f'runs:/{run_id}/model')
        cand_acc = accuracy_score(eval_y, candidate.predict(eval_X))
        # First model auto-promotes; later ones must beat the incumbent.
        try:
            prod = mlflow.sklearn.load_model(f'models:/{MODEL_NAME}/Production')
            prod_acc = accuracy_score(eval_y, prod.predict(eval_X))
        except Exception:
            prod_acc = -1.0
        if cand_acc <= prod_acc:
            return f'HOLD: {cand_acc:.3f} <= prod {prod_acc:.3f}'
        mv = mlflow.register_model(f'runs:/{run_id}/model', MODEL_NAME)
        mlflow.MlflowClient().transition_model_version_stage(
            MODEL_NAME, mv.version, 'Production', archive_existing_versions=True)
        return f'PROMOTED v{mv.version} (acc {cand_acc:.3f})'

Phase 2 — Feature Completion

Phase 2 adds the online half: the serving endpoint that loads the Production model by stage and logs predictions, and the drift monitor that reads those logs. This completes the feature set by connecting the offline spine to live traffic and observation, the serving component turns the registered model into a callable service, and the monitor turns its prediction log into a drift verdict. You will build a FastAPI app that loads by stage reference (never a hard-coded file), validates inputs, serves predictions, and appends each request to a prediction log, plus a monitor that compares the logged live features against the training reference. This phase makes the model observable in production, the prerequisite for closing the loop in Phase 3.

Analogy🏏Cricket
🏏 Think of it like cricket: with the squad selected, you now field the team and station analysts to watch every ball, the match management puts the chosen eleven into play, and the analysts log each delivery against the expected patterns. Just as match management fields whoever currently holds each role rather than a fixed name, serving loads whoever holds the Production stage rather than a fixed file. Just as the analysts logging every ball create the record that later reveals a developing weakness, prediction logging creates the record the monitor reads for drift. Just as fielding without observation would leave problems unseen, serving without monitoring would leave drift invisible. The insight is that putting the model into play and watching it closely are two halves of the same step, neither complete without the other.
python
# Phase 2: serving by stage with prediction logging + a drift monitor.
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
import mlflow, csv, os
from scipy import stats

MODEL_NAME = 'PlayerFormClassifier'
PRED_LOG = 'serving/prediction_log.csv'
app = FastAPI(title='Cricket Form Service')

_model = mlflow.sklearn.load_model(f'models:/{MODEL_NAME}/Production')  # by STAGE, once

class Innings(BaseModel):
    batting_average: float = Field(..., ge=0, le=120)
    strike_rate: float = Field(..., ge=0, le=400)
    boundary_pct: float = Field(..., ge=0, le=1)

def _log_prediction(f: Innings, pred: int):
    os.makedirs(os.path.dirname(PRED_LOG), exist_ok=True)
    new = not os.path.exists(PRED_LOG)
    with open(PRED_LOG, 'a', newline='') as fh:
        w = csv.writer(fh)
        if new: w.writerow(['batting_average', 'strike_rate', 'boundary_pct', 'prediction'])
        w.writerow([f.batting_average, f.strike_rate, f.boundary_pct, pred])

@app.post('/predict')
async def predict(f: Innings):
    try:
        pred = int(_model.predict([[f.batting_average, f.strike_rate, f.boundary_pct]])[0])
        _log_prediction(f, pred)               # record for the monitor
        return {'in_form': bool(pred)}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get('/healthz')
async def healthz(): return {'status': 'alive'}
@app.get('/readyz')
async def readyz(): return {'status': 'ready'}

class DriftMonitor:
    def __init__(self, reference_features):
        self.reference = list(reference_features)
    def drift_detected(self, log_path=PRED_LOG, min_n=100) -> bool:
        rows = list(csv.DictReader(open(log_path)))
        if len(rows) < min_n:
            return False
        live = [float(r['batting_average']) for r in rows[-500:]]
        return stats.ks_2samp(self.reference, live).pvalue < 0.05

Phase 3 — Polish & Production Readiness

Phase 3 closes the loop and hardens the system: the retraining trigger that consumes the drift verdict and launches a new tracked run routed back through the registry gate, plus the error handling and an end-to-end test that proves the whole cycle works. This is where the pipeline becomes production-ready rather than a collection of parts, the trigger makes the system self-renewing, and the test verifies that data flows correctly from raw data through training, promotion, serving, monitoring, and back to retraining. You will wire the trigger to fire on drift or accumulated data, ensure failures in any component are caught and surfaced rather than silently breaking the loop, and write a test that runs one full turn of the cycle and asserts a model ends up served and a drift signal can trigger a retrain.

Analogy🏏Cricket
🏏 Think of it like cricket: the final step in building a season-long operation is the post-match review that feeds the next selection, plus the contingency plans for when things go wrong mid-tour, so the whole cycle sustains itself and survives setbacks. Just as the review converts match observations into the next round of selection, the trigger converts drift signals into the next round of training. Just as a touring side needs plans for injuries and poor form so one failure does not derail the season, the pipeline needs error handling so one component failure does not break the loop. Just as a dry-run of the whole operation before the season confirms every handoff works, an end-to-end test confirms the whole pipeline flows. The insight is that production readiness is the loop closing on itself plus the resilience to keep running when parts fail.
python
# Phase 3: retraining trigger closing the loop + error handling + an end-to-end test.
import logging
logging.basicConfig(level=logging.INFO)
log = logging.getLogger('pipeline')

class RetrainTrigger:
    def __init__(self, trainer, registry, monitor, data):
        self.trainer, self.registry, self.monitor, self.data = trainer, registry, monitor, data
    def maybe_retrain(self, new_label_count=0, threshold=300):
        try:
            drift = self.monitor.drift_detected()
        except FileNotFoundError:
            drift = False                         # no predictions yet -> no drift
        if not (drift or new_label_count >= threshold):
            return 'no retrain needed'
        reason = 'drift' if drift else 'data_volume'
        log.info('Retraining triggered by %s', reason)
        try:
            run_id = self.trainer.train_and_track()
            X, y = self.data.training_set()       # eval on a held slice in practice
            result = self.registry.promote_if_better(run_id, X[-100:], y[-100:])
            return f'retrained ({reason}); registry: {result}'
        except Exception as e:                    # a failure must not break the loop
            log.error('Retraining failed: %s', e)
            return f'retrain FAILED: {e}'

# --- End-to-end test: one full turn of the cycle ---
def test_end_to_end():
    data = DataLayer()
    trainer = Trainer(data)
    registry = Registry()
    # 1) Train + promote the first model (auto-promotes as there is no incumbent).
    run_id = trainer.train_and_track()
    X, y = data.training_set()
    promo = registry.promote_if_better(run_id, X[-100:], y[-100:])
    assert 'PROMOTED' in promo, promo
    # 2) A Production model is now loadable by STAGE (serving contract holds).
    prod = mlflow.sklearn.load_model(f'models:/{MODEL_NAME}/Production')
    assert prod.predict([[53.6, 131.0, 0.61]])[0] in (0, 1)
    # 3) The retrain trigger runs without error and respects the gate.
    monitor = DriftMonitor(data.reference_features())
    trigger = RetrainTrigger(trainer, registry, monitor, data)
    out = trigger.maybe_retrain(new_label_count=500)
    assert 'retrained' in out or 'no retrain' in out, out
    print('END-TO-END OK:', promo, '|', out)

if __name__ == '__main__':
    test_end_to_end()

Evaluation Rubric

  • Lineage integrity: every served model traces back through the registry to a tracked run and a specific data version (25%).
  • Promotion gating: candidates are compared against the incumbent and only promoted when better, with the first model auto-promoting cleanly (15%).
  • Serving correctness: the API loads by stage reference, validates inputs with a typed schema, and exposes working health and readiness endpoints (15%).
  • Prediction logging and monitoring: requests are logged and the drift monitor returns a correct pass/fail verdict against a fixed reference (15%).
  • Loop closure: the retraining trigger fires on drift or data volume and routes a new candidate back through the registry gate (15%).
  • Resilience and reproducibility: component failures are caught and surfaced, and a fixed seed plus recorded data version make runs reproducible (15%).

Extension Challenges: (1) Replace the file-based prediction log with a proper store and add a held-out, time-based evaluation set so promotion is judged on data the model never trained on, not the training set. (2) Add shadow deployment to the serving layer so a newly promoted Staging model runs on mirrored traffic before it is moved to Production, integrating Module 4's safe-rollout techniques. (3) Wire the whole pipeline into an orchestrator (Airflow or Prefect) so the loop runs on a schedule with the drift check as a sensor, turning your script into a genuinely automated, observable production pipeline.

Submit your capstone project

Checking submission status…
Lesson 20 of 35
0% complete