Practice — Build a Reusable Production Pipeline — Data Analysis & Feature Engineering

What You'll Build

In this exercise you will build a complete, production-ready, reusable machine-learning pipeline for player-performance prediction, incorporating every element from this module: a ColumnTransformer with custom transformers, joblib serialisation with a model card, a DVC-ready project structure, and a pytest test suite. This is the culmination of the pipelines module — not a single concept in isolation but all of them assembled into one self-contained, tested, reproducible ML artifact that could be handed to a deployment engineer and deployed without further engineering. By the end you will have a pipeline that takes raw player data and delivers predictions, with the full quality, reproducibility, and testability infrastructure that production systems require.

Analogy🏏Cricket

🏏 Think of it like cricket: Building this pipeline is like a head analyst codifying the entire pre-match preparation routine into a single reusable playbook — pitch inspection, opposition profiling, matchup analysis, and threat assessment — that can be run before any match against any opponent. Just as the playbook turns scattered preparation habits into one repeatable, complete routine, your EDA pipeline turns scattered exploration steps into one repeatable engine. Just as a good playbook ensures no aspect of preparation is forgotten before any match, your pipeline ensures no aspect of exploration is skipped on any dataset. The insight is that codifying the full exploration workflow into a reusable tool is what makes thorough EDA fast, consistent, and complete every time.

Prerequisites

Python 3.10 or later with NumPy, pandas, scikit-learn, and joblib installed.
Mastery of Pipeline and ColumnTransformer from Lesson 25.
Command of custom transformers from Lesson 26 and joblib serialisation from Lesson 27.
Understanding of DVC data versioning concepts from Lesson 28 and pytest testing from Lesson 29.
Familiarity with the full feature engineering and selection pipeline from the preceding modules.

Setup & Project Structure

You will create a project with a clean directory structure that mirrors professional MLOps conventions: src for the pipeline code, tests for the test suite, models for serialised artifacts, and data for versioned data files. This structure exists because professional ML projects are maintained by teams over time, and a predictable, convention-following directory layout makes the project immediately navigable to any team member, while the separation of concerns between code, tests, and artifacts enables independent versioning and CI/CD automation.

Analogy🏏Cricket

🏏 Think of it like cricket: a smart side doesn't build a brand-new practice routine for every opponent — it develops a reusable set of net drills and fitness protocols that can be pointed at any upcoming team, then layers opponent-specific prep on top. That is exactly why you separate a reusable EDA engine module, holding your general analysis functions, from the dataset-specific script that aims them at the housing data: build the engine once and you can explore any future dataset with it, just as a well-designed net session works against any tour. Just as a coach fixes the bowling-machine settings and pitch so today's session can be repeated identically tomorrow, you install dependencies into an isolated virtual environment and seed every source of randomness — including the outlier detection — so the whole analysis is reproducible. Just as a structured training ground keeps drills organised and repeatable, a clean project structure keeps your engine and script cleanly divided. The payoff: a disciplined setup you can reuse and trust match after match.

bash

# Create the production-ready project structure
mkdir cricket_ml_pipeline && cd cricket_ml_pipeline
python -m venv venv && source venv/bin/activate
pip install numpy pandas scikit-learn joblib pytest

# Standard MLOps directory layout
mkdir -p src tests models data

# Files
touch src/__init__.py src/pipeline.py src/train.py
touch tests/__init__.py tests/test_pipeline.py

echo "Project structure mirrors professional MLOps conventions."
echo "src/ = pipeline code, tests/ = test suite, models/ = artifacts, data/ = versioned data"

Step 1 — Foundation

Step 1 builds the foundation pipeline in src/pipeline.py, combining a ColumnTransformer with a custom transformer inside a Pipeline. This is the foundation because it defines the reusable, version-controlled pipeline object that every other step uses — the trained serialised model, the tests, and the DVC pipeline all reference this single source of truth. Keeping the pipeline definition in one importable module ensures the same pipeline is used for training, testing, and serving, preventing divergence.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 1 is like the opening pitch-and-conditions inspection — establishing the basic facts of the playing surface before any tactical planning. Just as the pitch inspection must come first because every tactic depends on the conditions, the structure and quality checks must come first because every analysis depends on understanding them. Just as a misread pitch ruins the game plan, missed type errors or missingness ruin the analysis. The insight is that the foundational structure-and-quality inspection is the bedrock the whole exploration stands on.

python

# src/pipeline.py
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

class PercentileClipper(BaseEstimator, TransformerMixin):
    """Custom: clip to training percentile bounds (outlier-robust preprocessing)."""
    def __init__(self, lower_pct=2, upper_pct=98):
        self.lower_pct = lower_pct; self.upper_pct = upper_pct

    def fit(self, X, y=None):
        self.lower_ = np.percentile(X, self.lower_pct, axis=0)
        self.upper_ = np.percentile(X, self.upper_pct, axis=0)
        return self

    def transform(self, X):
        return np.clip(X, self.lower_, self.upper_)

def build_pipeline(C=1.0):
    """Return an unfitted production pipeline. C is tunable via GridSearchCV."""
    numeric = Pipeline([
        ("clip",   PercentileClipper()),                         # custom
        ("impute", SimpleImputer(strategy="median")),
        ("scale",  StandardScaler()),
    ])
    categorical = Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
    ])
    preprocessor = ColumnTransformer([
        ("num", numeric, make_column_selector(dtype_include="number")),
        ("cat", categorical, make_column_selector(dtype_include="object")),
    ])
    return Pipeline([
        ("prep",  preprocessor),
        ("model", LogisticRegression(C=C, max_iter=500, random_state=42)),
    ])

Step 2 — Core Logic

Step 2 builds the training and evaluation script in src/train.py that loads data, cross-validates, fits the final pipeline, and saves the serialised artifact with its model card. This is the analytical core because it is the script that runs during DVC's pipeline execution to produce the trained artifact, connecting the data source, the pipeline definition, and the output artifact in one reproducible step.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 2 is like profiling every single player in both squads with the metrics appropriate to their role — batting stats for batsmen, bowling figures for bowlers — so each is understood individually before matchups are considered. Just as each player gets the right kind of profile, each variable gets a type-appropriate summary. Just as profiling every player ensures none is overlooked before the matchup analysis, profiling every variable ensures none is skipped before relationship analysis. The insight is that complete, type-appropriate univariate profiling is the analytical core that everything downstream builds on.

python

# src/train.py
import numpy as np, pandas as pd, joblib, json, sklearn
from datetime import datetime
from pathlib import Path
from sklearn.model_selection import cross_val_score, train_test_split
from src.pipeline import build_pipeline

def train_and_save(data_path="data/players.csv", model_path="models/pipeline.pkl",
                   card_path="models/model_card.json"):
    df = pd.read_csv(data_path)
    X = df.drop("is_star", axis=1)
    y = df["is_star"]
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
    pipe = build_pipeline(C=1.0)
    cv_auc = cross_val_score(pipe, Xtr, ytr, cv=5, scoring="roc_auc").mean()
    pipe.fit(Xtr, ytr)
    test_auc = pipe.score(Xte, yte)
    # Save pipeline and model card together
    Path(model_path).parent.mkdir(exist_ok=True)
    joblib.dump(pipe, model_path, compress=3)
    card = {"version":"1.0", "saved_at":datetime.now().isoformat(),
            "cv_auc":round(cv_auc,4), "test_auc":round(test_auc,4),
            "n_features":X.shape[1], "feature_names":list(X.columns),
            "sklearn_version":sklearn.__version__}
    Path(card_path).write_text(json.dumps(card, indent=2))
    print(f"Saved pipeline. CV AUC={cv_auc:.3f}, Test AUC={test_auc:.3f}")
    return pipe, card

if __name__ == "__main__":
    train_and_save()

Step 3 — Integration & Enhancement

Step 3 writes the pytest test suite in tests/test_pipeline.py and defines the DVC pipeline stage, completing the reusable pipeline by adding automated quality assurance and reproducibility tracking. You will write the four critical tests (output shape, label validity, missing-value handling, unseen-category handling) as fixtures and test functions, and define the dvc.yaml stage that connects the data source to the trained model output, making the whole workflow reproducible with dvc repro.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 3 is like completing the preparation by analysing matchups between players, flagging anomalous performances, and turning the whole survey into a concrete game plan with specific instructions. Just as the matchup analysis and threat flags feed into an actionable plan, the bivariate and outlier analysis feed into the feature-engineering plan. Just as scattered observations are useless without a plan, EDA findings are useless without being made actionable. The insight is that integrating relationships and anomalies and converting everything into a concrete plan is what completes the pipeline and delivers EDA's real value.

python

# tests/test_pipeline.py
import pytest, numpy as np, pandas as pd
from src.pipeline import build_pipeline

@pytest.fixture(scope="module")
def fitted_pipeline():
    rng = np.random.default_rng(42)
    n = 300
    X = pd.DataFrame({"strike_rate": rng.normal(120,25,n),
                       "fitness": rng.normal(70,15,n),
                       "role": rng.choice(["bat","bowl","allround"], n)})
    y = (X["strike_rate"] > 120).astype(int)
    pipe = build_pipeline()
    pipe.fit(X, y)
    return pipe, X, y

def test_output_shape(fitted_pipeline):
    pipe, X, _ = fitted_pipeline
    assert pipe.predict(X).shape == (len(X),)

def test_valid_binary_labels(fitted_pipeline):
    pipe, X, _ = fitted_pipeline
    assert set(pipe.predict(X)).issubset({0, 1})

def test_handles_missing(fitted_pipeline):
    pipe, X, _ = fitted_pipeline
    X_miss = X.copy(); X_miss.loc[0, "fitness"] = float("nan")
    assert pipe.predict(X_miss).shape[0] == len(X_miss)

def test_handles_unseen_category(fitted_pipeline):
    pipe, X, _ = fitted_pipeline
    X_new = X.copy(); X_new.loc[0, "role"] = "wicketkeeper"
    assert pipe.predict(X_new).shape[0] == len(X_new)

def test_custom_clipper_uses_training_bounds(fitted_pipeline):
    pipe, X, _ = fitted_pipeline
    clipper = pipe["prep"].named_transformers_["num"]["clip"]
    assert hasattr(clipper, "lower_"), "Clipper should have learned lower_ from training"
    assert hasattr(clipper, "upper_"), "Clipper should have learned upper_ from training"

Step 4 — Testing & Verification

Run the full test suite and verify all tests pass, confirming the production pipeline is correct on all critical dimensions. Then review the test output and the model card to confirm the pipeline is ready for deployment. A green test suite plus a complete model card is the definition of a production-ready ML artifact.

Analogy🏏Cricket

🏏 Think of it like cricket: before the real series you play a full dress-rehearsal warm-up match to confirm all your preparation actually holds up under match conditions — and that is exactly what running the complete EDA pipeline on the housing data does. Just as you glance at the scoreboard to check it reads a sensible total and the right number of players are accounted for, you confirm the structure and quality checks report sensible shapes and plausible missingness. Just as you verify each player is listed in their correct role — batsman, bowler, keeper — you check the univariate profiles correctly classify and summarise every variable. Just as a warm-up reveals which opposition threats correlate most with danger, the relationship analysis should surface the features most strongly tied to valuation, and the outlier detection should flag genuinely reasonable freak cases, not nonsense. And just as you review the footage afterward to be sure nothing looked broken, you verify the whole output is coherent and actionable. The payoff: you trust the pipeline before it matters.

bash

# Run the test suite from the project root
# pytest tests/ -v

# Expected output:
# tests/test_pipeline.py::test_output_shape                 PASSED
# tests/test_pipeline.py::test_valid_binary_labels          PASSED
# tests/test_pipeline.py::test_handles_missing              PASSED
# tests/test_pipeline.py::test_handles_unseen_category      PASSED
# tests/test_pipeline.py::test_custom_clipper_uses_training_bounds  PASSED
# 5 passed in <time>

# This confirms:
# 1. Pipeline produces correct output shapes and labels.
# 2. Edge cases (missing, unseen categories) handled without crashing.
# 3. Custom transformer is stateful and learned its bounds from training.
#
# After training:
# python -m src.train  ->  models/pipeline.pkl + models/model_card.json
# These two files are the deployable artifact.
echo "pytest tests/ -v  ->  5 tests pass, confirming production readiness."
echo "src.train  ->  models/pipeline.pkl + models/model_card.json"

Warning: A green test suite does not guarantee the pipeline is correct — only that it satisfies the tested contracts. If the tests are weak or missing critical cases, a pipeline can pass all tests while still failing in production on data it was never tested with. Write tests that cover the real-world edge cases your production data will present: the distributions shift, new categories appear, columns occasionally go missing. Strong tests are the ones that catch the bugs you actually encounter in production, not just the ones that are easy to write.

Extension Challenge: Extend the project by adding a CI/CD configuration file (GitHub Actions or GitLab CI) that automatically runs pytest on every push, so the test suite is always enforced rather than manually run. As a stretch goal, add a performance regression test that loads the saved pipeline, runs it on a validation set, and fails if the AUC drops below the threshold recorded in the model card, ensuring that any new training run that degrades performance is caught before deployment.

A production-ready ML pipeline combines a ColumnTransformer with custom transformers, serialisation with a model card, and a full test suite.
A single importable pipeline module (src/pipeline.py) is the single source of truth that training, testing, and serving all reference.
The training script connects data, pipeline definition, and serialised artifact in one reproducible step that DVC can orchestrate.
The test suite covers the four critical contracts: output shape, label validity, edge-case handling, and custom transformer statefulness.
The paired pipeline file and model card are the deployable artifact; the DVC pipeline makes their production reproducible.
A green test suite confirms the tested contracts are satisfied, not that the pipeline is correct for all possible inputs.
All these elements together define the professional ML delivery standard: correct, tested, reproducible, and deployable.