Practice — Build a Data Cleaning Pipeline

What You'll Build

In this exercise you will build a complete, reusable data-cleaning pipeline that takes a messy raw dataset and produces a clean, model-ready one, applying every preprocessing technique from this module in the correct order. The dataset is a deliberately dirty cricket-player records file with missing values, inconsistent category labels, duplicates, mixed scales, and class imbalance. You will implement leakage-safe handling for each problem — diagnosing and imputing missing data, normalising and resolving inconsistent categories, removing duplicates before splitting, encoding categoricals, scaling numerics, and rebalancing classes inside the training fold. This consolidates the entire cleaning module into one tool that mirrors exactly how a data scientist transforms raw data into a trustworthy modelling input, with leakage prevention designed in at every step.

Prerequisites

• Python 3.10 or later with NumPy, pandas, scikit-learn, and imbalanced-learn installed.
• Mastery of missing-data mechanisms and imputation from Lesson 7.
• Command of categorical encoding from Lesson 8 and feature scaling from Lesson 9.
• Understanding of imbalance handling from Lesson 10 and duplicate detection from Lesson 11.
• Familiarity with scikit-learn pipelines and the fit-on-train discipline.

Setup & Project Structure

You will create a project with a cleaning-pipeline module and a script that runs a messy dataset through it. Separating the reusable pipeline from the dataset-specific script lets you apply the same cleaning to any future raw data. Install the dependencies into a virtual environment and seed all randomness so the cleaning, including resampling, is reproducible. The pipeline will be built with scikit-learn and imbalanced-learn components so that the leakage-prevention discipline is enforced structurally rather than relying on manual care.

# Create the project
mkdir cricket_cleaning && cd cricket_cleaning
python -m venv venv
source venv/bin/activate          # Windows: venv\Scripts\activate
pip install numpy pandas scikit-learn imbalanced-learn

# Project files
touch cleaning_pipeline.py run_cleaning.py

# Verify
python -c "import numpy, pandas, sklearn, imblearn; print('Cleaning stack ready')"

Step 1 — Foundation

Step 1 builds the foundation: the pre-split cleaning steps that must happen on the whole dataset before any train-test split — consistency normalisation, entity resolution, and duplicate removal. This is the foundation because these operations fix the structural integrity of the data and must precede splitting, since duplicates spanning a split leak information and inconsistent categories fragment the data regardless of split. Getting this stage right ensures the split itself operates on coherent, de-duplicated records.

# cleaning_pipeline.py
import pandas as pd

def clean_structure(df):
    """Pre-split fixes: normalise categories, resolve entities, drop duplicates."""
    df = df.copy()
    # Normalise inconsistent category labels (case, whitespace)
    for col in df.select_dtypes("object").columns:
        df[col] = df[col].str.strip().str.lower()
    # Entity resolution: map known variants to canonical forms
    team_map = {"csk": "chennai", "chennai super kings": "chennai",
                "mi": "mumbai", "mumbai indians": "mumbai"}
    if "team" in df.columns:
        df["team"] = df["team"].replace(team_map)
    # Remove exact duplicates BEFORE any split (prevents leakage)
    before = len(df)
    df = df.drop_duplicates().reset_index(drop=True)
    print(f"Removed {before - len(df)} duplicate rows; {len(df)} remain")
    return df

if __name__ == "__main__":
    raw = pd.read_csv("messy_players.csv")
    cleaned = clean_structure(raw)
    print(f"Teams after resolution: {cleaned['team'].unique()}")

Step 2 — Core Logic

Step 2 builds the core transformation pipeline using scikit-learn's ColumnTransformer to apply leakage-safe imputation, encoding, and scaling to the right columns. This is the analytical core because it bundles the per-column transformations into a single fit-on-train object, structurally guaranteeing that imputation statistics, encoder categories, and scaling parameters are learned only from training data. Building it as a ColumnTransformer ensures numeric and categorical columns each receive their appropriate treatment within one coherent, leakage-proof component.

# cleaning_pipeline.py (continued)
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.preprocessing import OneHotEncoder, RobustScaler

def build_transformer(numeric_cols, categorical_cols):
    """Leakage-safe per-column transformations bundled together."""
    numeric = Pipeline([
        ("impute", KNNImputer(n_neighbors=5)),       # relationship-preserving
        ("scale", RobustScaler()),                    # outlier-resistant
    ])
    categorical = Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
    ])
    return ColumnTransformer([
        ("num", numeric, numeric_cols),
        ("cat", categorical, categorical_cols),
    ])

Step 3 — Integration & Enhancement

Step 3 integrates the transformer with class rebalancing into a full imbalanced-learn pipeline and assembles the end-to-end cleaning-and-modelling flow. You will combine the ColumnTransformer with SMOTE and a classifier in an imbalanced-learn pipeline, which ensures resampling happens only on training folds, then split the data with a grouped split to prevent entity leakage. This integration completes the pipeline by connecting per-column cleaning to leakage-safe rebalancing within one object, so that every leakage risk — from imputation, scaling, encoding, and resampling — is structurally prevented in a single coherent flow.

# cleaning_pipeline.py (continued)
from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE
from sklearn.ensemble import RandomForestClassifier

def build_full_pipeline(numeric_cols, categorical_cols):
    """End-to-end: clean -> rebalance -> model, all leakage-safe per fold."""
    transformer = build_transformer(numeric_cols, categorical_cols)
    return ImbPipeline([
        ("clean", transformer),                       # impute, encode, scale
        ("balance", SMOTE(random_state=42)),          # resample TRAIN fold only
        ("model", RandomForestClassifier(random_state=42)),
    ])
# Note: SMOTE runs during fit (training) but is skipped during predict (test),
# and all transformers refit on each training fold -> zero leakage by design.

Step 4 — Testing & Verification

Now you will run the complete pipeline on the messy dataset and verify it produces clean, leakage-free results. Apply the structural cleaning, perform a grouped split to keep each player's records together, fit the full pipeline via cross-validation, and confirm that the cleaning resolved categories and removed duplicates, that no missing values remain after transformation, and that the cross-validated scores are honest because every step refit per fold. Sensible, stable cross-validated performance with no leakage verifies the pipeline works end to end.

# run_cleaning.py
import pandas as pd
from sklearn.model_selection import GroupShuffleSplit, cross_val_score
from cleaning_pipeline import clean_structure, build_full_pipeline

raw = pd.read_csv("messy_players.csv")

# Step 1: structural cleaning BEFORE split
df = clean_structure(raw)

# Define columns and target
numeric_cols = ["runs", "strike_rate", "fitness"]
categorical_cols = ["team", "role"]
X = df[numeric_cols + categorical_cols]
y = df["is_star"]
groups = df["player_id"]

# Grouped split keeps each player entirely in train OR test (no entity leakage)
gss = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=42)
train_idx, test_idx = next(gss.split(X, y, groups=groups))

# Full leakage-safe pipeline, evaluated by cross-validation
pipe = build_full_pipeline(numeric_cols, categorical_cols)
scores = cross_val_score(pipe, X.iloc[train_idx], y.iloc[train_idx],
                         cv=5, scoring="f1")
print(f"Cross-validated F1: {scores.mean():.3f} (+/- {scores.std():.3f})")
pipe.fit(X.iloc[train_idx], y.iloc[train_idx])
test_f1 = pipe.score(X.iloc[test_idx], y.iloc[test_idx])
print(f"Held-out test accuracy: {test_f1:.3f}")
print("Every step refit per fold -> honest, leakage-free evaluation.")

• A cleaning pipeline transforms messy raw data into model-ready data through a correctly ordered sequence of steps.
• Structural fixes that learn no parameters — consistency normalisation and duplicate removal — belong before the train-test split.
• Parameter-learning steps — imputation, scaling, encoding, resampling — must live inside a pipeline that refits per training fold.
• A ColumnTransformer applies leakage-safe, column-appropriate treatments to numeric and categorical columns within one object.
• An imbalanced-learn pipeline ensures resampling happens only on training folds, skipped during prediction on test data.
• Grouped splitting keeps each entity's records together, preventing the entity leakage that fragmented splits cause.
• Designing leakage prevention into the pipeline structurally is safer than relying on manual care at each step.