Practice — Evaluate Models with Cross-Validation — Machine Learning with Scikit-learn

What You'll Build

In this exercise you will build a rigorous model evaluation framework that applies every concept from this module: paradigm identification, bias-variance diagnosis, a proper three-way split with stratification, cross-validation of full preprocessing pipelines, and metric selection aligned with the business objective. The scenario is a cricket talent-scouting tool — predicting whether a junior player will become a professional cricketer within five years based on their early performance metrics. This is a real-world imbalanced classification problem (few juniors make it to professional level), which demands stratified splits, AUC-based evaluation, and careful threshold tuning to balance false positives against false negatives.

Analogy🏏Cricket

🏏 Think of it like cricket: Predicting house prices is structurally identical to predicting match scores: the features (area, location, age) play the role of match conditions (pitch, weather, opposition), and the goal is to build a pipeline that makes the most accurate predictions while being interpretable enough for stakeholders (buyers, sellers, regulators) to trust. Just as a match-score predictor must pass diagnostic checks to ensure it models pitch and weather effects correctly, a house-price model must pass residual diagnostics before being used for financial decisions.

Prerequisites

Python 3.10 or later with NumPy, pandas, and scikit-learn installed.
Familiarity with supervised learning paradigms from Lesson 01.
Understanding of bias-variance trade-off and its diagnostics from Lesson 02.
Mastery of train/validation/test split strategies from Lesson 03.
Command of cross-validation variants and their implementation from Lesson 04.
Understanding of evaluation metrics and their alignment with business objectives from Lesson 05.

Setup & Dataset

You will generate a synthetic junior-cricketer dataset with realistic imbalance and evaluate multiple models against it. Install the required packages, seed for reproducibility, and inspect the class distribution before any modelling — the class imbalance informs every subsequent decision, from split strategy to metric choice to threshold setting.

Analogy🏏Cricket

🏏 Think of it like cricket: before you evaluate any young cricketer, you first survey the raw talent pool and notice how imbalanced it is — genuine future stars are rare, while journeymen fill most of the squad. Just as a synthetic junior-cricketer dataset is generated with realistic imbalance and a fixed seed for reproducibility, you assemble your scouting cohort under identical conditions so any two selectors comparing notes see the very same players. Just as you inspect the class distribution before choosing a selection method — noticing only a handful of prospects are true match-winners — you inspect the imbalance before modelling, because that scarcity dictates everything downstream: how you split the trials, which statistic you trust, and where you set the bar for selection. The payoff: understanding the shape of your talent pool first means every later decision, from split strategy to metric choice to threshold, rests on solid ground rather than a false picture.

python

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification

# Synthetic junior-cricketer dataset: 10% make professional level (realistic imbalance)
X, y = make_classification(
    n_samples=1000, n_features=15, n_informative=8, n_redundant=4,
    weights=[0.90, 0.10], flip_y=0.02, random_state=42
)
feature_names = [
    "batting_avg", "strike_rate", "bowling_economy", "fielding_runs",
    "fitness_score", "mental_score", "coachability", "match_wins",
    "team_contribution", "technique_score", "consistency", "aggression",
    "game_iq", "form_trend", "injury_history"
]
df = pd.DataFrame(X, columns=feature_names)
df["professional"] = y

print(f"Dataset: {df.shape}")
print(f"Class distribution:\n{df['professional'].value_counts(normalize=True).round(3)}")
print(f"Positive (professional) rate: {y.mean():.1%}")
print("Imbalanced dataset -> stratified splits and AUC-based metrics are required.")

Step 1 — Foundation: Proper Data Splitting

Step 1 establishes the correct data split before any modelling. With an imbalanced target, stratification is mandatory. The test set is locked away immediately and will be used only once at the very end. All modelling decisions — feature selection, hyperparameter tuning, model comparison — will use cross-validation on the training set only.

Analogy🏏Cricket

🏏 Think of it like cricket: laying down a proper data split is like sealing your final World Cup trial in an envelope before the season even starts. With rare match-winners in the pool, stratification is mandatory — you deliberately ensure the locked-away trial contains its fair share of genuine stars, just as you keep a proportional sprinkle of them in every practice round, so no round is stripped of the rare class. Just as a selector who peeks at the final trial to tweak his choices corrupts it into a meaningless formality, the test set is locked away immediately and touched only once at the very end. Every intermediate decision — which attributes to scout, how strict the selection bar, which coaching plan wins — is settled using rotation (cross-validation) on the practice rounds alone. The payoff: because the final trial stayed genuinely unseen, its verdict is an honest forecast of tournament form rather than a rehearsed illusion.

python

from sklearn.model_selection import train_test_split

X_trainval, X_test, y_trainval, y_test = train_test_split(
    df.drop("professional", axis=1).values, df["professional"].values,
    test_size=0.20, stratify=df["professional"].values, random_state=42
)

print(f"Train+val: {len(X_trainval)} samples  "
      f"({y_trainval.mean():.1%} positive)")
print(f"Test:      {len(X_test)} samples  "
      f"({y_test.mean():.1%} positive)")
print("Test set locked. No peeking until ALL decisions are finalised.")
print(f"Stratification confirmed: both splits preserve the ~10% positive rate.")

Step 2 — Core Logic: Cross-Validated Pipeline Evaluation

Step 2 builds the core evaluation logic: a function that cross-validates any sklearn-compatible pipeline on the training data and returns mean AUC-ROC, AUC-PR, and their standard deviations. Using both AUC-ROC and AUC-PR is essential for imbalanced data — AUC-ROC measures overall ranking quality, while AUC-PR specifically measures minority-class performance and is harder to inflate. The function cross-validates a full pipeline (preprocessing + model) to prevent leakage.

Analogy🏏Cricket

🏏 Think of it like cricket: the core evaluation engine is like a standard, repeatable trial that judges any candidate's full game — technique, temperament, the works — across rotating practice rounds and reports both his average and how much he wobbles round to round. Crucially it reports two statistics, because judging a rare-talent pool on one number lies. AUC-ROC is like overall ranking ability — can he generally sort the good deliveries from the bad across all conditions — while AUC-PR zooms in specifically on how well he handles the scarce, decisive match-winning moments, and is far harder to inflate when those moments are rare. Just as trusting only a batting average hides a player who feasts on weak bowling but folds against genuine pace, trusting only AUC-ROC hides poor minority-class performance. The payoff: a single reusable trial that fairly scores any coaching plan on both overall quality and its handling of the rare cases that actually decide matches.

python

from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

def evaluate_pipeline(pipeline, X, y, n_splits=5):
    cv = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=42)
    results = cross_validate(
        pipeline, X, y, cv=cv,
        scoring={"roc_auc": "roc_auc",
                 "avg_precision": "average_precision",  # AUC-PR
                 "f1": "f1"},
        return_train_score=True
    )
    return {
        "train_auc": results["train_roc_auc"].mean(),
        "val_auc":   results["test_roc_auc"].mean(),
        "val_auc_std": results["test_roc_auc"].std(),
        "val_pr_auc":  results["test_avg_precision"].mean(),
        "val_f1":      results["test_f1"].mean(),
        "bias_var_gap": results["train_roc_auc"].mean() - results["test_roc_auc"].mean(),
    }

def make_pipe(model):
    return Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
        ("model", model),
    ])

models = {
    "Logistic Regression": make_pipe(LogisticRegression(max_iter=500, C=1.0)),
    "Decision Tree (deep)": make_pipe(DecisionTreeClassifier(max_depth=None)),
    "Decision Tree (d=4)":  make_pipe(DecisionTreeClassifier(max_depth=4)),
    "Random Forest":        make_pipe(RandomForestClassifier(n_estimators=100, random_state=42)),
}
print(f"{'Model':30} | {'Train AUC':>10} | {'Val AUC':>10} | {'Val PR AUC':>11} | {'Diagnosis'}")
for name, pipe in models.items():
    r = evaluate_pipeline(pipe, X_trainval, y_trainval)
    if r["train_auc"] < 0.70:    diag = "High Bias"
    elif r["bias_var_gap"] > 0.15: diag = "High Variance"
    else:                          diag = "Balanced"
    print(f"{name:30} | {r['train_auc']:>10.3f} | {r['val_auc']:>10.3f} | "
          f"{r['val_pr_auc']:>11.3f} | {diag}")

Step 3 — Integration: Bias-Variance Diagnosis and Threshold Tuning

Step 3 selects the best model based on cross-validation, diagnoses its bias-variance position using a learning curve, and tunes the classification threshold to maximise F1 (or a domain-specific metric). In talent scouting, missing a future star (false negative) is more costly than an extra tryout invitation (false positive), so the threshold should be tuned to favour recall over precision.

Analogy🏏Cricket

🏏 Think of it like cricket: after the trials name your best coaching plan, you diagnose exactly why it wins and then set the selection bar deliberately. A learning curve is like watching whether a prospect keeps improving as he faces more net sessions — if he plateaus early and struggles even in practice, he is under-coached (high bias); if he crushes the nets but crumbles in fresh games, he is over-fitted to familiar bowling (high variance). Tuning the classification threshold is like deciding how strict the selection cut-off should be: in talent scouting, missing a future star (false negative) is far costlier than handing out one extra tryout (false positive), so you lower the bar to favour recall, catching more genuine prospects at the price of a few wasted invitations. Just as a wise selector diagnoses a player's flaw before adjusting his standards, you diagnose bias-variance before tuning the threshold — the payoff being a model tuned to the real cost of each mistake.

python

from sklearn.model_selection import learning_curve
from sklearn.metrics import precision_recall_curve, roc_auc_score, average_precision_score
import numpy as np

# Select best model by val AUC (Random Forest assumed)
best_pipe = make_pipe(RandomForestClassifier(n_estimators=100, random_state=42))

# LEARNING CURVE: diagnose bias vs variance
train_sizes, train_scores, val_scores = learning_curve(
    best_pipe, X_trainval, y_trainval,
    cv=StratifiedKFold(5, shuffle=True, random_state=42),
    scoring="roc_auc",
    train_sizes=np.linspace(0.1, 1.0, 6)
)
print("Learning curve (AUC):")
for sz, tr, va in zip(train_sizes, train_scores.mean(1), val_scores.mean(1)):
    gap = tr - va
    diagnosis = "High Variance (overfit)" if gap > 0.15 else "OK" if gap > 0.05 else "High Bias"
    print(f"  n={sz:4.0f}: train={tr:.3f}, val={va:.3f}, gap={gap:.3f} -> {diagnosis}")

# THRESHOLD TUNING using out-of-fold predictions
from sklearn.model_selection import cross_val_predict
y_oof_prob = cross_val_predict(best_pipe, X_trainval, y_trainval,
                               cv=StratifiedKFold(5, shuffle=True, random_state=42),
                               method="predict_proba")[:,1]
prec, rec, thr = precision_recall_curve(y_trainval, y_oof_prob)
f1 = 2*prec*rec/(prec+rec+1e-9)
best_thr = thr[f1[:-1].argmax()]
print(f"\nOptimal threshold (max F1 on OOF): {best_thr:.3f}")
print(f"At this threshold: precision={prec[f1[:-1].argmax()]:.3f}, "
      f"recall={rec[f1[:-1].argmax()]:.3f}")
print("Lower threshold -> higher recall (fewer future stars missed)")

Step 4 — Testing & Final Report

Now use the test set — exactly once — to report the final performance. Train the best model on the full training set, apply the tuned threshold, and compute all metrics. Compare the test performance against the CV estimate to check they are close, confirming the CV was an honest estimate. A large discrepancy signals the CV was somehow contaminated.

Analogy🏏Cricket

🏏 Think of it like cricket: the final report is the sealed World Cup trial opened exactly once. You take your chosen coaching plan, train it on the full practice pool, apply the strict-or-lenient selection bar you settled earlier, and record how the prospects actually perform on the unseen trial — no second attempts, no re-tuning. Then you compare this verdict against what the rotating practice rounds predicted: if the trial result closely matches the cross-validated estimate, it confirms your practice judgements were honest all along. Just as a large gap between a player's glowing practice reputation and a dismal trial exposes that the practice sessions were somehow rigged or leaked, a big discrepancy between test and CV performance signals the cross-validation was contaminated. The payoff: a single, uncorrupted final measurement that you can genuinely trust to forecast real-world performance, with a built-in consistency check that catches hidden leakage.

python

from sklearn.metrics import (roc_auc_score, average_precision_score,
                             precision_score, recall_score, f1_score, confusion_matrix)
import numpy as np

# Train final model on ALL trainval data
best_pipe.fit(X_trainval, y_trainval)

# TEST SET evaluation — one look only
y_test_prob = best_pipe.predict_proba(X_test)[:,1]
y_test_pred = (y_test_prob > best_thr).astype(int)

print("=" * 55)
print("FINAL TEST SET RESULTS (one-time, uncontaminated)")
print("=" * 55)
print(f"AUC-ROC:   {roc_auc_score(y_test, y_test_prob):.3f}")
print(f"AUC-PR:    {average_precision_score(y_test, y_test_prob):.3f}")
print(f"F1:        {f1_score(y_test, y_test_pred):.3f}")
print(f"Precision: {precision_score(y_test, y_test_pred):.3f}")
print(f"Recall:    {recall_score(y_test, y_test_pred):.3f}  (fraction of future stars caught)")
print(f"\nConfusion matrix:")
cm = confusion_matrix(y_test, y_test_pred)
print(f"  TN={cm[0,0]:3d}  FP={cm[0,1]:3d}")
print(f"  FN={cm[1,0]:3d}  TP={cm[1,1]:3d}")
print(f"\nCV AUC (training estimate): ~0.XXX  vs  Test AUC: {roc_auc_score(y_test, y_test_prob):.3f}")
print("Close agreement confirms the CV was an honest, non-contaminated estimate.")

Warning: The exercise design requires strict test-set discipline — compute the test-set metrics only in Step 4, after all decisions in Steps 1-3 are finalised. If you peek at the test set to guide threshold selection or model choice in Step 3, the test result in Step 4 is no longer an honest evaluation. If accidental peeking occurs, document it and generate a new random seed for the test split so the reported result remains honest.

Extension Challenge: Replace the default threshold-maximising F1 with a threshold that maximises expected monetary value using domain costs: assume each missed future star (false negative) costs the board INR 50 lakhs in future contracts, and each unnecessary trial invitation (false positive) costs INR 50,000. Compute the expected value at each threshold and find the business-optimal threshold, which will likely be much lower than the F1-optimal threshold, favouring high recall over high precision.

Imbalanced classification requires stratified splits, AUC-based metrics, and threshold tuning — accuracy alone is meaningless.
Cross-validate full pipelines (imputer + scaler + model) to prevent preprocessing leakage from test folds.
Both AUC-ROC and AUC-PR should be reported for imbalanced problems; AUC-PR is more sensitive to minority-class performance.
The learning curve diagnoses bias versus variance before tuning; diagnosing before acting prevents applying the wrong remedy.
Out-of-fold predictions provide an honest dataset for threshold tuning without touching the test set.
The test set is used exactly once at the end; close agreement between CV AUC and test AUC validates the evaluation.
Threshold selection is a business decision based on the relative cost of false positives and false negatives, not a fixed 0.5.