What You'll Build
In this exercise you will build a complete competition-grade ensemble pipeline — the kind that consistently wins Kaggle competitions — for a cricket player performance prediction task. Starting from raw features, you will engineer meaningful domain features, train five diverse base models, use out-of-fold stacking with a meta-learner, apply Optuna hyperparameter tuning to the best base model, and produce a final leaderboard submission with comprehensive performance analysis.
The scenario is predicting whether a cricket player will be selected for the national team next season based on their current performance metrics. This is a binary classification task with moderate class imbalance — selection is a competitive process, so only the top performers are selected. The entire pipeline mirrors a real Kaggle competition workflow: EDA, feature engineering, diverse modelling, stacking, Optuna tuning, and final evaluation with business interpretation.
Analogy🏏Cricket
🏏 Think of it like cricket: Predicting house prices is structurally identical to predicting match scores: the features (area, location, age) play the role of match conditions (pitch, weather, opposition), and the goal is to build a pipeline that makes the most accurate predictions while being interpretable enough for stakeholders (buyers, sellers, regulators) to trust. Just as a match-score predictor must pass diagnostic checks to ensure it models pitch and weather effects correctly, a house-price model must pass residual diagnostics before being used for financial decisions.
Prerequisites
- Python 3.10 or later with NumPy, pandas, scikit-learn, and optuna installed.
- Mastery of gradient boosting from Lesson 20 and stacking from Lesson 22.
- Solid understanding of OOF stacking and leakage prevention from Lesson 22.
- Command of Optuna Bayesian hyperparameter tuning from Lesson 23.
- Familiarity with the full evaluation framework from Module 1 (AUC-ROC, AUC-PR, stratified CV).
Setup and Dataset
Generate a realistic cricket player selection dataset with non-linear interactions and meaningful feature engineering opportunities. The dataset has 15 raw features including batting and bowling statistics, fitness scores, and team context variables. Understanding the data structure before any modelling determines which features to engineer and which algorithms to include in the ensemble.
Analogy🏏Cricket
🏏 Think of it like cricket: setting up the house-price dataset is like surveying a lopsided pool of player valuations before you start modelling. Prices are right-skewed — a handful of superstar mansions tower over a mass of ordinary homes, just as a few marquee players' fees dwarf the rest. The area-price link is non-linear, larger houses costing disproportionately more per square metre, like elite all-rounders commanding a premium that balloons rather than adds up. And features are multicollinear, several overlapping stats telling the same story. Applying an immediate log transform to the skewed target is like switching valuations to a compressed scale so the giants no longer distort the whole picture, exactly as analysts log-transform lopsided fees before comparing them. Just as a scout first tames a skewed valuation pool into a fair, comparable scale before ranking anyone, you log the target upfront. The payoff: a well-conditioned dataset where later modelling isn't dominated by a few extreme outliers.
python
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(42)
n = 1200
# Raw cricket performance features
batting_avg = rng.normal(35, 15, n).clip(0, 100)
strike_rate = rng.normal(130, 30, n).clip(50, 250)
bowling_avg = rng.normal(30, 12, n).clip(10, 80)
bowling_econ = rng.normal(7.5, 2, n).clip(3, 15)
catches_year = rng.integers(0, 30, n).astype(float)
fitness_score = rng.uniform(60, 100, n)
age = rng.uniform(18, 38, n)
exp_years = (age - 18 + rng.normal(0, 1, n)).clip(0, 20)
home_avg = batting_avg + rng.normal(5, 8, n)
away_avg = batting_avg - rng.normal(5, 8, n)
recent_form = rng.normal(0, 1, n) # standardised recent performance
team_rank = rng.integers(1, 20, n).astype(float)
injury_flag = rng.choice([0, 1], n, p=[0.85, 0.15])
contract_type = rng.choice([0, 1, 2], n, p=[0.6, 0.3, 0.1]) # 0=domestic, 1=ipl, 2=int
wickets_year = rng.integers(0, 60, n).astype(float)
# True selection: non-linear, interaction-heavy
log_odds = (
0.05*batting_avg + 0.01*strike_rate - 0.03*bowling_avg
- 0.1*bowling_econ + 0.1*fitness_score/100*batting_avg/50 # interaction
- 0.15*injury_flag + 0.1*recent_form - 0.02*(age - 28)**2 # quadratic age
+ 0.3*(contract_type == 2) - 0.3 + rng.normal(0, 0.7, n)
)
prob_selected = 1 / (1 + np.exp(-log_odds))
y = (prob_selected > 0.7).astype(int) # only top ~15% selected
feature_names = ["batting_avg","strike_rate","bowling_avg","bowling_econ","catches_year",
"fitness_score","age","exp_years","home_avg","away_avg","recent_form",
"team_rank","injury_flag","contract_type","wickets_year"]
X_raw = np.column_stack([batting_avg, strike_rate, bowling_avg, bowling_econ, catches_year,
fitness_score, age, exp_years, home_avg, away_avg, recent_form,
team_rank, injury_flag, contract_type, wickets_year])
print(f"Dataset: {X_raw.shape}")
print(f"Selection rate: {y.mean():.1%} (imbalanced: only top ~15% selected)")
print(f"\nFeature correlations with selection:")
corrs = [np.corrcoef(X_raw[:,i], y)[0,1] for i in range(X_raw.shape[1])]
for name, corr in sorted(zip(feature_names, corrs), key=lambda x: abs(x[1]), reverse=True)[:5]:
print(f" {name:20}: {corr:+.3f}")
Step 1 — Foundation: Feature Engineering
Engineer domain-meaningful features before any modelling: the all-rounder score combining batting and bowling, form-fitness interaction, location advantage, age-experience ratio, bowling contribution relative to team need, and recent vs historical comparison. Feature engineering is the highest-value step — these derived features directly expose the patterns that drive selection.
Analogy🏏Cricket
🏏 Think of it like cricket: raw stats rarely tell a selector the real story, so a shrewd coach engineers his own measures. Just as you combine batting and bowling numbers into a single all-rounder rating to value a Stokes over a pure specialist, you build an all-rounder score from batting and bowling columns. Just as a selector weighs current form against fitness — a player in touch but carrying a niggle is a different bet — you craft a form-fitness interaction feature. Just as home advantage, an age-versus-experience balance, and a bowler's value relative to what the attack lacks all sway a pick, you derive location advantage, age-experience ratio, and bowling contribution relative to team need, plus recent-versus-career comparisons. Engineering these domain features before any modelling is the highest-value move — they expose the selection patterns directly, so even a simple model can read them, giving you the biggest jump in accuracy for the least effort.
python
import numpy as np
def engineer_features(X_raw, feature_names):
X = X_raw.copy()
col = dict(zip(feature_names, range(len(feature_names))))
batting = X[:, col["batting_avg"]]
sr = X[:, col["strike_rate"]]
bowl_avg = X[:, col["bowling_avg"]]
bowl_eco = X[:, col["bowling_econ"]]
fitness = X[:, col["fitness_score"]]
age = X[:, col["age"]]
exp = X[:, col["exp_years"]]
home = X[:, col["home_avg"]]
away = X[:, col["away_avg"]]
recent = X[:, col["recent_form"]]
injury = X[:, col["injury_flag"]]
wickets = X[:, col["wickets_year"]]
# Domain-driven engineered features
allrounder_score = (batting / 40) + (25 / bowl_avg.clip(min=1)) # combined value
form_fitness = recent * (fitness / 100) # interaction
location_adv = home - away # home/away gap
age_peak_dist = (age - 27) ** 2 # quadratic age effect
batting_efficiency= batting * (sr / 130) # risk-adjusted batting
bowling_value = wickets / bowl_eco.clip(min=0.1) # economy-weighted wickets
experience_ratio = exp / age.clip(min=1) # early vs late career
injury_penalty = 1 - 0.3 * injury # injury adjustment
new_features = np.column_stack([
allrounder_score, form_fitness, location_adv,
age_peak_dist, batting_efficiency, bowling_value,
experience_ratio, injury_penalty
])
new_names = ["allrounder_score","form_fitness","location_adv",
"age_peak_dist","batting_efficiency","bowling_value",
"experience_ratio","injury_penalty"]
return np.hstack([X_raw, new_features]), feature_names + new_names
X_fe, all_features = engineer_features(X_raw, feature_names)
print(f"Features after engineering: {X_fe.shape[1]} ({X_fe.shape[1] - X_raw.shape[1]} new)")
# Stratified split
from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(X_fe, y, test_size=0.20, stratify=y, random_state=42)
print(f"Train: {len(X_tr)}, Test: {len(X_te)}")
print("Feature engineering done — domain features expose the selection signal directly.")
Step 2 — Core Logic: Diverse Base Models + OOF Stacking
Train five diverse base models using OOF cross-validation and stack them with a regularised logistic regression meta-learner. The diversity spans linear (logistic regression), tree ensemble (random forest), sequential ensemble (gradient boosting), maximum margin (SVM), and distance-based (KNN) — each capturing different aspects of the selection pattern.
Analogy🏏Cricket
🏏 Think of it like cricket: a strong selection panel deliberately includes different kinds of judges, not five clones. Just as you'd want a stats analyst (linear logistic regression), a scout who reads many angles (random forest), a coach who corrects mistakes ball by ball (gradient boosting), a specialist who focuses on the borderline calls (max-margin SVM), and a comparer who judges by similar past players (distance-based KNN), you assemble five diverse base models — each catching a different facet of the pattern. Just as you'd never let a judge grade a player he already coached — that's biased — you use out-of-fold cross-validation so each model predicts only on players it never trained on. Then, just as a chief selector weighs the panel's honest verdicts into one call, a regularised logistic-regression meta-learner stacks the OOF predictions. Diversity plus unbiased stacking beats any single expert's blind spots.
python
from sklearn.model_selection import StratifiedKFold, cross_val_predict, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import roc_auc_score, average_precision_score
import numpy as np
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
base_models = {
"LogReg": Pipeline([("sc", StandardScaler()), ("m", LogisticRegression(C=0.1, max_iter=500, class_weight="balanced"))]),
"RandForest": RandomForestClassifier(n_estimators=300, class_weight="balanced", n_jobs=-1, random_state=42),
"GradBoost": GradientBoostingClassifier(n_estimators=200, learning_rate=0.05, max_depth=4, subsample=0.8, random_state=42),
"SVM": Pipeline([("sc", StandardScaler()), ("m", SVC(C=1.0, gamma="scale", class_weight="balanced", probability=True))]),
"KNN": Pipeline([("sc", StandardScaler()), ("m", KNeighborsClassifier(n_neighbors=15))]),
}
# OOF predictions (leak-free training for meta-learner)
oof_preds_tr = np.zeros((len(X_tr), len(base_models)))
te_preds = np.zeros((len(X_te), len(base_models)))
print(f"{'Model':14} | {'Base CV AUC':>12} | {'Base CV PR':>11}")
for i, (name, model) in enumerate(base_models.items()):
oof_preds_tr[:, i] = cross_val_predict(model, X_tr, y_tr, cv=cv, method="predict_proba")[:,1]
base_auc = roc_auc_score(y_tr, oof_preds_tr[:,i])
base_pr = average_precision_score(y_tr, oof_preds_tr[:,i])
fold_te = []
for tr_idx, val_idx in cv.split(X_tr, y_tr):
model.fit(X_tr[tr_idx], y_tr[tr_idx])
fold_te.append(model.predict_proba(X_te)[:,1])
te_preds[:, i] = np.mean(fold_te, axis=0)
print(f"{name:14} | {base_auc:>12.3f} | {base_pr:>11.3f}")
# META-LEARNER on OOF predictions
meta = LogisticRegression(C=0.01, max_iter=500)
meta.fit(oof_preds_tr, y_tr)
stack_te_prob = meta.predict_proba(te_preds)[:,1]
print(f"\nNaive average: test AUC = {roc_auc_score(y_te, te_preds.mean(axis=1)):.3f} "
f"PR = {average_precision_score(y_te, te_preds.mean(axis=1)):.3f}")
print(f"Stacking: test AUC = {roc_auc_score(y_te, stack_te_prob):.3f} "
f"PR = {average_precision_score(y_te, stack_te_prob):.3f}")
Step 3 — Integration: Optuna Tuning of Best Base Model
Identify the best base model from Step 2 and tune it with Optuna, then update the stacking ensemble with the tuned model. The tuned model's OOF predictions replace its untuned counterpart in the stacking matrix, and the meta-learner is retrained on the updated predictions.
Analogy🏏Cricket
🏏 Think of it like cricket: after watching the panel work, you spot which judge reads the game best and invest extra coaching in that one. Just as a captain fine-tunes his strike bowler's field and length rather than reworking the whole attack, you take the strongest base model from Step 2 and tune its hyperparameters with Optuna — searching combinations the way you'd trial line and length in the nets. Just as that bowler then returns to the eleven and his sharper spells replace his old ones in the match plan, the tuned model's out-of-fold predictions replace its untuned column in the stacking matrix. And just as the captain re-reads his whole strategy once his best weapon improves, you retrain the meta-learner on the updated predictions. Targeted tuning of the one component that matters most lifts the entire ensemble, without wasting effort polishing pieces that were already good enough.
python
import optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.ensemble import GradientBoostingClassifier
cv_tune = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
def objective_gb(trial):
params = {
"n_estimators": trial.suggest_int("n_estimators", 50, 400),
"learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
"max_depth": trial.suggest_int("max_depth", 2, 7),
"subsample": trial.suggest_float("subsample", 0.6, 1.0),
"min_samples_leaf":trial.suggest_int("min_samples_leaf", 5, 40),
"max_features": trial.suggest_float("max_features", 0.4, 1.0),
}
gb = GradientBoostingClassifier(**params, random_state=42)
return cross_val_score(gb, X_tr, y_tr, cv=cv_tune, scoring="roc_auc").mean()
study = optuna.create_study(direction="maximize",
sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective_gb, n_trials=25)
print(f"Optuna best GradBoost AUC: {study.best_value:.3f}")
print(f"Best params: {study.best_params}")
# Update stacking with tuned model
tuned_gb = GradientBoostingClassifier(**study.best_params, random_state=42)
from sklearn.model_selection import cross_val_predict
oof_tuned = cross_val_predict(tuned_gb, X_tr, y_tr, cv=cv, method="predict_proba")[:,1]
oof_preds_updated = oof_preds_tr.copy()
oof_preds_updated[:, 2] = oof_tuned # replace GradBoost column
# Retrain meta-learner on updated OOF
meta_updated = LogisticRegression(C=0.01, max_iter=500)
meta_updated.fit(oof_preds_updated, y_tr)
fold_te_tuned = []
for tr_idx, val_idx in cv.split(X_tr, y_tr):
tuned_gb.fit(X_tr[tr_idx], y_tr[tr_idx])
fold_te_tuned.append(tuned_gb.predict_proba(X_te)[:,1])
te_preds_updated = te_preds.copy()
te_preds_updated[:, 2] = np.mean(fold_te_tuned, axis=0)
stack_updated_prob = meta_updated.predict_proba(te_preds_updated)[:,1]
print(f"Stacking with tuned GradBoost: test AUC = {roc_auc_score(y_te, stack_updated_prob):.3f}")
Step 4 — Final Report and Submission
Produce the final competition-grade report: all metrics on the test set, threshold analysis aligned to the selection context (recall is critical — missing a genuine talent is costly), feature importance summary, and a plain-language selection narrative explaining the model's top factors. This report mirrors what a Kaggle top-solution writeup includes.
Analogy🏏Cricket
🏏 Think of it like cricket: the innings is over, and now you file the match report the selection board actually reads. Just as a report lists every stat — runs, strike rate, economy — you present all test-set metrics. Just as a selector sets the bar knowing that overlooking a genuine talent costs a series, you tune your decision threshold to favour recall, because missing a real prospect is far worse than a false alarm. Just as a scout ranks which factors — form, fitness, conditions — drove each verdict, you summarise feature importance. And just as the best writeups end with a plain-language rationale a director can follow, you close with a narrative naming the model's top selection factors. This competition-grade report mirrors a Kaggle top-solution writeup: it doesn't just declare a winner, it proves, in numbers and words, exactly why the model chose whom — which is what turns a good score into a trusted one.
python
from sklearn.metrics import (roc_auc_score, average_precision_score,
precision_score, recall_score, f1_score,
confusion_matrix, precision_recall_curve)
import numpy as np
# Final predictions from best stacking ensemble
y_prob_final = stack_updated_prob
# Business-optimal threshold: selection recall is critical
# Missing a genuine talent (FN) = lost national-team potential (high cost)
# Wrong selection (FP) = wasted trial camp spot (lower cost)
prec, rec, thresholds = precision_recall_curve(y_te, y_prob_final)
# Threshold that achieves at least 80% recall
min_recall = 0.80
valid_mask = rec[:-1] >= min_recall
if valid_mask.any():
best_thr = thresholds[valid_mask][np.argmax(prec[:-1][valid_mask])]
else:
best_thr = 0.3 # fallback
y_pred_final = (y_prob_final >= best_thr).astype(int)
cm = confusion_matrix(y_te, y_pred_final)
tn, fp, fn, tp = cm.ravel()
print("=" * 65)
print("FINAL TEST SET RESULTS — CRICKET SELECTION MODEL")
print("=" * 65)
print(f"AUC-ROC: {roc_auc_score(y_te, y_prob_final):.3f}")
print(f"AUC-PR: {average_precision_score(y_te, y_prob_final):.3f}")
print(f"Threshold: {best_thr:.3f} (set for >= 80% recall)")
print(f"Precision: {precision_score(y_te, y_pred_final):.3f} (fraction of selected who are genuine talents)")
print(f"Recall: {recall_score(y_te, y_pred_final):.3f} (fraction of genuine talents correctly identified)")
print(f"F1: {f1_score(y_te, y_pred_final):.3f}")
print(f"\nConfusion matrix:")
print(f" Genuine talents correctly selected (TP): {tp}")
print(f" Genuine talents missed (FN): {fn}")
print(f" Non-talents incorrectly selected (FP): {fp}")
print(f" Non-talents correctly excluded (TN): {tn}")
print(f"\nModel identifies {recall_score(y_te, y_pred_final):.0%} of genuine national-level talents.")
print(f"Of all recommended for trial, {precision_score(y_te, y_pred_final):.0%} are genuine selections.")
Warning: In the exercise the test set is evaluated once after all decisions (feature engineering choices, model selection, stacking architecture, and Optuna tuning) are finalised. If you experimented with the test set during development — checking its AUC to guide feature engineering or model choices — the reported test performance is optimistically biased and should be re-evaluated on fresh data. In a real competition, the public leaderboard can serve as this guard, but the true test is the private leaderboard that you only see once the competition ends.
Extension Challenge: Extend the stacking ensemble with a third level — train a second meta-learner on the first meta-learner's OOF predictions combined with the original base model OOF predictions. Multi-level stacking (level 0 → level 1 → level 2) can extract additional signal from the combination of base model and first-level stacker predictions, though it requires careful OOF generation at each level and strong regularisation at the upper levels to prevent overfitting the tiny meta-prediction matrices.
- Competition-grade pipelines combine feature engineering, diverse base models, OOF stacking, and Optuna tuning in a leakage-free workflow.
- Domain-meaningful feature engineering (allrounder score, form-fitness interaction, location advantage) exposes the signal that base models alone miss.
- Five diverse base models (linear, tree ensemble, sequential ensemble, kernel, distance) provide complementary predictions for the meta-learner.
- OOF stacking requires k-fold generation for training the meta-learner and k-model averaging for test predictions.
- Optuna with 25-50 trials efficiently finds near-optimal gradient boosting hyperparameters; updating the stacking ensemble with tuned predictions improves the final model.
- Business-aligned threshold selection (ensuring high recall for genuine talents) produces more actionable predictions than the default 0.5.
- The final test set is evaluated exactly once after all pipeline decisions are locked — this is the true measure of generalisation.