Foundation Practice: ML Experiment Pipeline

What You'll Build

In this exercise you will build a reproducible ML experiment pipeline that ties together everything from Module 1: versioned data, tracked experiments, and verifiable reproducibility. The pipeline trains a classifier that predicts whether a cricketer is currently in form from recent batting and bowling features, runs several configurations as tracked experiments, records each run's parameters and metrics, and persists the winning model as a versioned artifact. You will then prove the pipeline is reproducible by rerunning a configuration and confirming byte-identical results. By the end you will have a small but complete MLOps foundation, the kind of skeleton every production system is built on, where data, code, and model are jointly pinned and any past result can be recovered, compared, and trusted rather than vaguely remembered.

Analogy🏏Cricket

🏏 Think of it like cricket: a selection committee does not pick a squad on a hunch; they run structured trial matches under recorded conditions, log every player's scores, compare candidates on identical criteria, and keep the records so a selection can be justified later. Just as the trial matches are your tracked experiment runs, each player's logged scores are your run metrics. Just as the committee compares candidates on the same pitch to be fair, you compare model configurations on the same data. Just as a defensible selection can be reproduced from the records if challenged, your pipeline can reproduce any run from its logged inputs. The insight is that disciplined, recorded trials, not gut feel, are what make both squad selection and model selection defensible.

Prerequisites

Python 3.10 or newer installed, with pip available for installing the required packages.
Comfort with basic Python: functions, dictionaries, and reading a small dataset from disk.
Conceptual understanding of the ML lifecycle, reproducibility, and experiment tracking from lessons 01 and 02.
Familiarity with the idea of pinning random seeds and recording parameters and metrics per run.
A terminal where you can create a project folder, run scripts, and install MLflow locally.

Setup & Project Structure

You will create a self-contained project folder with a clear separation between data, source code, and tracked outputs, mirroring how real MLOps projects are organised. The data lives under a data directory, the pipeline logic under src, and experiment tracking is handled by MLflow writing to a local mlruns store. Keeping these concerns in separate folders matters because it makes the project's structure self-documenting and lets you reason about data versus code versus results independently, exactly the discipline that scales from a toy pipeline to a production platform. Install the two dependencies, MLflow for tracking and scikit-learn for the model, then lay out the directories.

Analogy🏏Cricket

🏏 Think of it like cricket: a well-run academy never dumps kit, players, and match records into one heap; it keeps the practice ground, the coaching staff's playbook, and the scorers' logbook in separate, clearly labelled areas so anyone can find what they need and nothing gets mixed up. Just as you separate the data directory, the src pipeline logic, and the mlruns tracking store, the academy separates its pitches, its coaching manuals, and its performance ledgers. Just as keeping these concerns apart makes a project's structure obvious at a glance, a tidy academy lets a new coach walk in and immediately know where drills, plans, and records live. Just as MLflow writes runs to a local store beside, not inside, the code, the scorers keep the logbook outside the coaching manual so results never overwrite strategy. The payoff: a clean layout means a stranger, or you in six months, can pick up the project and understand exactly how data becomes a model.

bash

# Create the project skeleton and install dependencies.
mkdir -p cricket-form-pipeline/{data,src,outputs}
cd cricket-form-pipeline

python -m venv .venv
source .venv/bin/activate        # on Windows: .venv\Scripts\activate

pip install mlflow==2.* scikit-learn==1.* pandas==2.*

# Resulting structure:
# cricket-form-pipeline/
# |-- data/
# |   `-- players.csv          # versioned input dataset
# |-- src/
# |   |-- prepare.py           # Step 1: load + feature engineering
# |   |-- train.py             # Step 2: train + track a run
# |   `-- pipeline.py          # Step 3: sweep configs + select best
# |-- outputs/                 # best model artifact lands here
# `-- mlruns/                  # MLflow local tracking store (auto-created)

echo 'Project skeleton ready.'

Step 1 — Foundation

Step 1 builds the data foundation and feature engineering, the deterministic base every later stage depends on. You will create a small cricket dataset and a pure feature function that derives model inputs from raw statistics. The concept behind this step is that reproducibility starts at the data layer: if feature engineering is a pure function of a fixed, versioned dataset, then identical inputs always yield identical features, removing one whole class of hidden variability. Writing the data version into the file itself, and keeping the transformation free of randomness, is what lets every downstream run be anchored to a known starting point that you can recover and reason about precisely.

Analogy🏏Cricket

🏏 Think of it like cricket: before any trial match, the groundsman prepares one agreed pitch and the same set of balls, so every candidate is judged on identical conditions rather than a surface that changes underfoot. Just as the fixed pitch is your versioned dataset, the standard ball is your deterministic feature function. Just as changing the pitch mid-trial would make scores incomparable, a feature function with hidden randomness would make runs incomparable. Just as recording the pitch and ball used lets you recreate the conditions, recording the data version lets you recreate the inputs. The insight is that fair, repeatable comparison demands a fixed, documented starting surface.

python

# src/prepare.py  -- Step 1: versioned data + deterministic features.
import csv, os

DATA_VERSION = '[email protected]'   # pinned identity of this dataset

RAW_PLAYERS = [
    # name, matches, runs, wickets
    ('Rohit Sharma', 260, 10709, 8),
    ('Virat Kohli', 295, 13848, 4),
    ('Shubman Gill', 47, 2271, 0),
    ('Jasprit Bumrah', 89, 60, 149),
    ('Ravindra Jadeja', 197, 2756, 220),
    ('Tail Ender A', 30, 180, 2),
    ('Out Of Form B', 40, 410, 1),
    ('Fringe Player C', 22, 290, 0),
]

def write_dataset(path='data/players.csv'):
    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, 'w', newline='') as f:
        w = csv.writer(f)
        w.writerow(['name', 'matches', 'runs', 'wickets'])
        w.writerows(RAW_PLAYERS)
    return path

def engineer_features(path='data/players.csv'):
    """Pure, deterministic: same file -> same features, every time."""
    X, y, names = [], [], []
    for r in csv.DictReader(open(path)):
        matches = max(int(r['matches']), 1)
        batting_average = int(r['runs']) / matches
        wicket_rate = int(r['wickets']) / matches
        X.append([round(batting_average, 3), round(wicket_rate, 3)])
        y.append(1 if batting_average > 35 or wicket_rate > 0.8 else 0)  # 'in form'
        names.append(r['name'])
    return X, y, names

if __name__ == '__main__':
    write_dataset()
    X, y, names = engineer_features()
    print(f'Prepared {len(X)} players from {DATA_VERSION}')
    for n, f, label in zip(names, X, y):
        print(f'  {n:>16}  feats={f}  in_form={label}')

Step 2 — Core Logic

Step 2 builds the core training and tracking logic: a function that trains a model for a given configuration and logs the run to MLflow. This is the heart of the pipeline because it is where parameters become recorded inputs and accuracy becomes a recorded metric, joined to a versioned model artifact under a single run ID. By pinning the random seed inside the configuration and logging the data version as a parameter, every run becomes a self-describing record that can be compared with its siblings and reproduced exactly later. This step turns training from an ephemeral act into a durable, queryable experiment, which is the whole point of tracking.

Analogy🏏Cricket

🏏 Think of it like cricket: when a candidate bats in a trial, an official scorer records the conditions they batted in, the pitch, the bowling, and their exact score, all under one innings entry. Just as the recorded conditions are your logged params, the score is your logged metric, and the innings entry is the MLflow run. Just as the same conditions and the same player should reproduce a comparable knock, the same params and seed reproduce the same accuracy. Just as a scorer who noted only the score but not the conditions leaves selectors guessing, logging a metric without its params leaves you unable to explain a result. The insight is that a result is only useful when recorded together with the conditions that produced it.

python

# src/train.py  -- Step 2: train one configuration and track it in MLflow.
import mlflow, mlflow.sklearn
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
from prepare import engineer_features, DATA_VERSION

mlflow.set_experiment('cricket-form-pipeline')

def run_experiment(config: dict) -> dict:
    X, y, _ = engineer_features()
    with mlflow.start_run(run_name=config['name']) as run:
        mlflow.log_params({
            'max_depth': config['max_depth'],
            'criterion': config['criterion'],
            'seed': config['seed'],
            'data_version': DATA_VERSION,     # reproducibility anchor
        })
        model = DecisionTreeClassifier(
            max_depth=config['max_depth'],
            criterion=config['criterion'],
            random_state=config['seed'],
        )
        model.fit(X, y)
        acc = accuracy_score(y, model.predict(X))   # toy: train==eval for demo
        mlflow.log_metric('accuracy', acc)
        mlflow.sklearn.log_model(model, 'model')
        return {'run_id': run.info.run_id, 'accuracy': acc, 'name': config['name']}

if __name__ == '__main__':
    result = run_experiment(
        {'name': 'dt-depth3', 'max_depth': 3, 'criterion': 'gini', 'seed': 1983})
    print('Logged run:', result)

Step 3 — Integration & Enhancement

Step 3 brings the pieces together into a sweep that runs several configurations, compares their tracked metrics, and promotes the best model into the outputs folder as a versioned artifact. This integration step is where the pipeline becomes genuinely useful: instead of manually trying configurations and remembering which won, you execute the whole comparison programmatically, query MLflow for the top run, and persist the winner with a clear name. Selecting by logged metric rather than by intuition is the enhancement that mirrors how production systems choose models, on recorded evidence, and it leaves behind a complete trail from every candidate to the chosen one.

Analogy🏏Cricket

🏏 Think of it like cricket: after a round of trial matches, the committee lays every candidate's recorded scores side by side, picks the top performer on the agreed metric, and formally names them to the squad while keeping the rest on record. Just as laying out all scorecards is your sweep across configs, choosing the highest scorer is your selection by logged accuracy. Just as the named player is promoted while others remain in the pool for recall, the best model is promoted while others stay tracked. Just as the selection memo cites the scores that justified it, your promotion cites the winning run ID. The insight is that evidence-based selection across recorded candidates produces a defensible, reversible choice.

python

# src/pipeline.py  -- Step 3: sweep configs, compare, promote the best.
import os, shutil
import mlflow
from mlflow.tracking import MlflowClient
from train import run_experiment

CONFIGS = [
    {'name': 'dt-depth2', 'max_depth': 2, 'criterion': 'gini', 'seed': 1983},
    {'name': 'dt-depth3', 'max_depth': 3, 'criterion': 'gini', 'seed': 1983},
    {'name': 'dt-entropy3', 'max_depth': 3, 'criterion': 'entropy', 'seed': 1983},
]

def run_sweep():
    results = [run_experiment(cfg) for cfg in CONFIGS]
    best = max(results, key=lambda r: r['accuracy'])
    print('\nLeaderboard:')
    for r in sorted(results, key=lambda r: -r['accuracy']):
        flag = '  <-- best' if r['run_id'] == best['run_id'] else ''
        print(f"  {r['name']:>12}  acc={r['accuracy']:.3f}{flag}")
    promote(best)
    return best

def promote(best):
    os.makedirs('outputs', exist_ok=True)
    client = MlflowClient()
    src_uri = f"runs:/{best['run_id']}/model"
    local = mlflow.artifacts.download_artifacts(src_uri)
    dest = 'outputs/best_model'
    if os.path.exists(dest):
        shutil.rmtree(dest)
    shutil.copytree(local, dest)
    print(f"\nPromoted {best['name']} (run {best['run_id'][:8]}) -> {dest}")

if __name__ == '__main__':
    run_sweep()

Step 4 — Testing & Verification

Now verify the two properties that define a working foundation: the pipeline runs end to end and produces a tracked leaderboard plus a promoted model, and it is reproducible, meaning a rerun of the same configuration yields the same accuracy. Run the pipeline, inspect the leaderboard, then launch the MLflow UI to confirm every run is recorded with its params, metric, and model artifact. Finally rerun a single config twice and confirm the accuracy is identical, demonstrating that the pinned seed and versioned data delivered on the reproducibility promise.

Analogy🏏Cricket

🏏 Think of it like cricket: before trusting a match plan you verify two things, that it actually works start to finish under real conditions and produces a clear winner, and that it is repeatable, run the same trial again in the same conditions and you get the same result rather than a fluke. Just as you confirm the pipeline runs end to end and produces a tracked leaderboard plus a promoted model, a coach confirms the full session runs from warm-up to selection and yields a ranked shortlist with a clear pick. Just as you rerun the same configuration and check it yields the same accuracy, a coach reruns the identical trial and expects the same player to top the table, proving it was skill, not chance. Just as the MLflow UI records every run with its params, metric, and artifact, the scorebook records every trial so the result can be inspected and defended. The payoff: verifying both correctness and reproducibility, in a pipeline or a trial, means you trust the winner because it holds up when repeated.

bash

# Run the pipeline and verify reproducibility.
cd cricket-form-pipeline && source .venv/bin/activate

python src/prepare.py        # Step 1: build data + features
python src/pipeline.py       # Steps 2-3: sweep, compare, promote

# Expected (illustrative) output:
# Leaderboard:
#    dt-depth3  acc=1.000  <-- best
#  dt-entropy3  acc=1.000
#    dt-depth2  acc=0.875
# Promoted dt-depth3 (run a1b2c3d4) -> outputs/best_model

# Inspect every tracked run in the browser:
mlflow ui --port 5000        # open http://localhost:5000

# Reproducibility check: same config twice -> identical accuracy.
python -c "from src.train import run_experiment; \
cfg={'name':'repro','max_depth':3,'criterion':'gini','seed':1983}; \
a=run_experiment(cfg)['accuracy']; b=run_experiment(cfg)['accuracy']; \
print('reproducible:', a==b, a, b)"
# Expected: reproducible: True 1.0 1.0

Warning: The most common error here is omitting random_state (the seed) when constructing the model, then being baffled when two runs of the same config produce different accuracies and the reproducibility check prints False. Many scikit-learn estimators use randomness internally; without a fixed seed, identical params still yield different fits. Always thread the seed into the estimator and log it as a param so runs are genuinely reproducible.

Extension Challenge: Replace the in-script CONFIGS list with a params.yaml file and load it at runtime, then add a real train/test split so accuracy reflects generalisation rather than memorisation. For a harder stretch, persist a data-version hash alongside each run and add a guard that refuses to promote a model if its data version differs from the incumbent's, turning your toy pipeline into a genuine lineage-aware promotion gate.

A reproducible pipeline pins the data version, random seed, and parameters together so any past run can be recovered and compared.
Deterministic, pure feature engineering on a fixed dataset removes hidden variability and anchors every downstream run.
Tracking each run's params, metric, and model artifact under one run ID turns training into a durable, queryable experiment.
Selecting the best model by logged metric, not intuition, produces a defensible, reversible choice with a full candidate trail.
Always set and log the estimator's random_state, since unseeded randomness breaks reproducibility even with identical parameters.
Separating data, code, and tracked outputs into clear folders is the structural discipline that scales from a toy to a production platform.