What You'll Build
In this exercise you will build a comprehensive classification pipeline for customer churn prediction, applying every algorithm from this module: logistic regression as the interpretable baseline, a decision tree for rule discovery, a random forest as the main non-linear model, and an SVM as the high-margin classifier. You will compare them rigorously on stratified cross-validation, tune the best model's hyperparameters, interpret its decisions for a business audience, and produce a final report with threshold analysis aligned to the business cost of false positives versus false negatives.
The scenario is a telecommunications company trying to identify subscribers who will churn in the next month so the retention team can intervene before the cancellation. This is a canonical imbalanced binary classification problem where missing a churning customer (false negative) is more costly than incorrectly targeting a loyal one (false positive). Every modelling decision — metric choice, class weighting, threshold selection — will be driven by this business context.
Prerequisites
- Python 3.10 or later with NumPy, pandas, and scikit-learn installed.
- Mastery of logistic regression from Lesson 13.
- Understanding of decision trees and random forests from Lessons 14 and 15.
- Familiarity with SVM from Lesson 16 and KNN/Naive Bayes from Lesson 17.
- Command of cross-validation, AUC-ROC, AUC-PR, and threshold tuning from Module 1.
Setup and Dataset
Generate a realistic telecom churn dataset with typical characteristics: moderate imbalance (15% churn rate), a mix of numeric and categorical features, and a non-linear but learnable signal. Inspect the class distribution and feature correlations before splitting the data — understanding the data structure before any modelling prevents misspecified evaluation choices.
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(42)
n = 2000
# Telecom churn dataset
tenure_months = rng.integers(1, 72, n).astype(float)
monthly_charges = rng.uniform(20, 120, n)
contract_type = rng.choice([0, 1, 2], n, p=[0.5, 0.3, 0.2]) # 0=monthly, 1=1yr, 2=2yr
num_services = rng.integers(1, 9, n).astype(float)
tech_support = rng.choice([0, 1], n, p=[0.6, 0.4])
payment_method = rng.choice([0, 1, 2, 3], n) # 4 methods
senior_citizen = rng.choice([0, 1], n, p=[0.84, 0.16])
# True churn probability: non-linear interactions
log_odds_churn = (
-0.03 * tenure_months # longer tenure -> less likely to churn
+ 0.02 * monthly_charges # higher charges -> more likely
- 0.5 * contract_type # long-term contract -> less likely
- 0.1 * num_services # more services -> more locked in
- 0.3 * tech_support # tech support -> less likely
+ 0.15 * senior_citizen # seniors -> slightly more likely
+ rng.normal(0, 0.8, n)
)
prob_churn = 1 / (1 + np.exp(-log_odds_churn))
y = rng.binomial(1, prob_churn)
df = pd.DataFrame({
"tenure_months": tenure_months, "monthly_charges": monthly_charges,
"contract_type": contract_type, "num_services": num_services,
"tech_support": tech_support, "payment_method": payment_method,
"senior_citizen": senior_citizen, "churned": y,
})
print(f"Dataset: {df.shape}")
print(f"Churn rate: {y.mean():.1%} (imbalanced -> stratified splits, AUC-PR metric)")
print(f"\nFeature preview:")
print(df.describe().round(2))
Step 1 — Foundation: Split and Preprocessing Pipeline
Apply an immediate stratified train-test split and build the preprocessing pipeline for the mixed-type feature set. The pipeline handles numeric scaling and categorical one-hot encoding inside ColumnTransformer, ensuring all transformations are fit on training data only and applied consistently to both splits.
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
feature_cols = ["tenure_months","monthly_charges","contract_type","num_services",
"tech_support","payment_method","senior_citizen"]
numeric_cols = ["tenure_months","monthly_charges","num_services"]
categorical_cols = ["contract_type","tech_support","payment_method","senior_citizen"]
X = df[feature_cols].values
y = df["churned"].values
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.20, stratify=y, random_state=42)
print(f"Train: {len(X_tr)} (churn={y_tr.mean():.1%}), Test: {len(X_te)} (churn={y_te.mean():.1%})")
# Shared preprocessing: applies to all classifiers
numeric_idx = [feature_cols.index(c) for c in numeric_cols]
categorical_idx = [feature_cols.index(c) for c in categorical_cols]
preprocessor = ColumnTransformer([
("num", Pipeline([("imp", SimpleImputer(strategy="median")),
("sc", StandardScaler())]), numeric_idx),
("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), categorical_idx),
])
print("Preprocessing pipeline built: numeric (scale) + categorical (OHE).")
print("Will be fit on training fold only inside each cross-validation run.")
Step 2 — Core Logic: Train and Compare All Classifiers
Train all five classifiers using the shared preprocessing pipeline and five-fold stratified cross-validation. Report both AUC-ROC and AUC-PR for each, since the dataset is imbalanced and AUC-PR is the more informative metric for the minority churn class. Include the naive baseline (always predict non-churn) as the floor.
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
import numpy as np
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
classifiers = {
"Logistic Regression": LogisticRegression(C=1.0, max_iter=500, class_weight="balanced"),
"Decision Tree (d=5)": DecisionTreeClassifier(max_depth=5, min_samples_leaf=15,
class_weight="balanced", random_state=42),
"Random Forest": RandomForestClassifier(n_estimators=200, class_weight="balanced",
n_jobs=-1, random_state=42),
"SVM (RBF)": SVC(kernel="rbf", C=1.0, gamma="scale",
class_weight="balanced", probability=True),
"Naive Bayes": GaussianNB(),
}
print(f"{'Model':22} | {'AUC-ROC':>9} | {'AUC-PR':>9}")
print("-" * 47)
results = {}
for name, clf in classifiers.items():
pipe = Pipeline([("prep", preprocessor), ("clf", clf)])
roc = cross_val_score(pipe, X_tr, y_tr, cv=cv, scoring="roc_auc").mean()
pr = cross_val_score(pipe, X_tr, y_tr, cv=cv, scoring="average_precision").mean()
results[name] = (roc, pr, pipe)
print(f"{name:22} | {roc:>9.3f} | {pr:>9.3f}")
# Naive baseline
baseline_pr = y_tr.mean() # average precision of always-positive predictor
print(f"{'Naive baseline':22} | {'0.500':>9} | {baseline_pr:>9.3f}")
print("\nBest model by AUC-PR:")
best = max(results, key=lambda k: results[k][1])
print(f" {best}: AUC-PR = {results[best][1]:.3f}")
Step 3 — Integration: Tune Best Model and Interpret
Tune the random forest's key hyperparameters by grid search, then interpret the model for the business team. Compute feature importances, translate them into business language, and tune the classification threshold to maximise expected business value — assuming missing a churning customer costs INR 2,000 (revenue loss) while incorrectly targeting a loyal customer costs INR 200 (wasted retention offer).
from sklearn.model_selection import GridSearchCV, cross_val_predict
from sklearn.inspection import permutation_importance
from sklearn.metrics import precision_recall_curve
import numpy as np
# Tune Random Forest hyperparameters
param_grid = {
"clf__n_estimators": [200, 400],
"clf__max_features": ["sqrt", 0.5],
"clf__min_samples_leaf": [5, 15, 30],
}
rf_pipe = Pipeline([("prep", preprocessor),
("clf", RandomForestClassifier(class_weight="balanced",
n_jobs=-1, random_state=42))])
gs = GridSearchCV(rf_pipe, param_grid, cv=cv, scoring="average_precision", n_jobs=-1)
gs.fit(X_tr, y_tr)
print(f"Best params: {gs.best_params_}")
print(f"Best CV AUC-PR: {gs.best_score_:.3f}")
# Out-of-fold probabilities for threshold tuning
best_pipe = gs.best_estimator_
y_oof_prob = cross_val_predict(best_pipe, X_tr, y_tr, cv=cv, method="predict_proba")[:,1]
# BUSINESS-VALUE threshold tuning
# Cost of FN (missed churn): INR 2000 | Cost of FP (wasted offer): INR 200
prec, rec, thresholds = precision_recall_curve(y_tr, y_oof_prob)
n_test_approx = len(y_tr)
fn_cost, fp_cost = 2000, 200
# For each threshold: compute total cost
total_costs = []
for i, thr in enumerate(thresholds):
pred = (y_oof_prob >= thr).astype(int)
fn = ((pred==0) & (y_tr==1)).sum()
fp = ((pred==1) & (y_tr==0)).sum()
total_costs.append(fn*fn_cost + fp*fp_cost)
best_idx = np.argmin(total_costs)
best_thr = thresholds[best_idx]
print(f"\nBusiness-value optimal threshold: {best_thr:.3f}")
print(f"At this threshold: precision={prec[best_idx]:.3f}, recall={rec[best_idx]:.3f}")
print(f"Min total cost: INR {total_costs[best_idx]:,.0f}")
print(f"Default (0.5) cost: INR {total_costs[np.searchsorted(thresholds, 0.5)]:,.0f}")
Step 4 — Testing and Final Business Report
Evaluate the tuned model with the business-optimal threshold on the held-out test set. Report all metrics and translate the confusion matrix into business impact: how many churning customers were correctly identified (saved revenue), how many were missed (lost revenue), and how much retention budget was spent on non-churning customers (wasted offers).
from sklearn.metrics import (roc_auc_score, average_precision_score,
precision_score, recall_score, f1_score, confusion_matrix)
import numpy as np
# Final model trained on all training data
best_pipe.fit(X_tr, y_tr)
y_te_prob = best_pipe.predict_proba(X_te)[:,1]
y_te_pred = (y_te_prob >= best_thr).astype(int)
cm = confusion_matrix(y_te, y_te_pred)
tn, fp, fn, tp = cm.ravel()
print("=" * 60)
print("FINAL TEST SET RESULTS")
print("=" * 60)
print(f"AUC-ROC: {roc_auc_score(y_te, y_te_prob):.3f}")
print(f"AUC-PR: {average_precision_score(y_te, y_te_prob):.3f}")
print(f"Precision: {precision_score(y_te, y_te_pred):.3f}")
print(f"Recall: {recall_score(y_te, y_te_pred):.3f}")
print(f"F1: {f1_score(y_te, y_te_pred):.3f}")
# Business impact translation
actual_churners = tp + fn
correctly_caught = tp
missed_churners = fn
false_alarms = fp
print(f"\nBusiness Impact:")
print(f" Actual churners in test set: {actual_churners}")
print(f" Correctly identified (saved): {correctly_caught} "
f"(revenue saved: INR {correctly_caught*2000:,})")
print(f" Missed (lost revenue): {missed_churners} "
f"(loss: INR {missed_churners*2000:,})")
print(f" Loyal customers incorrectly targeted: {false_alarms} "
f"(wasted offers: INR {false_alarms*200:,})")
total_cost = missed_churners*2000 + false_alarms*200
total_cost_baseline = actual_churners*2000 # baseline: miss all churners
print(f"\nTotal cost at business threshold: INR {total_cost:,}")
print(f"Total cost without model (miss all): INR {total_cost_baseline:,}")
print(f"Model saves: INR {total_cost_baseline - total_cost:,} "
f"({(1 - total_cost/total_cost_baseline)*100:.1f}% reduction)")
Warning: The threshold selected using out-of-fold predictions on the training set may not be perfectly calibrated for the test set due to the distribution shift between the two. Always report the test-set cost at the business threshold alongside the threshold value, and note that in production the threshold may need periodic recalibration as customer behaviour evolves. Never fix the threshold permanently without a monitoring plan that detects when precision and recall drift away from the business-optimal operating point.
Extension Challenge: Add a cost-sensitive learning component by setting sample_weight in the classifier's fit call, giving each training example a weight proportional to the business cost of misclassifying it (churners weighted at 10x the weight of non-churners, reflecting the 2000/200 cost ratio). Compare the AUC-PR and business cost of the cost-sensitively-trained model against the class_weight='balanced' model and the business-threshold-tuned model. Which approach most effectively reduces the business cost on the test set?
- Imbalanced churn classification requires AUC-PR as the primary metric; AUC-ROC can be misleadingly high for the majority class.
- The preprocessing pipeline (ColumnTransformer + Pipeline) must be shared across all classifiers for a fair comparison.
- Random forests with class_weight='balanced' typically outperform logistic regression on non-linear churn patterns.
- Threshold selection based on business cost (FN cost vs FP cost) produces better business outcomes than F1-optimal or default 0.5 thresholds.
- Out-of-fold probabilities from cross_val_predict provide an honest, leakage-free dataset for threshold optimisation.
- The confusion matrix must be translated into business impact (revenue saved, wasted offers, missed churners) for stakeholder communication.
- Monitor threshold performance over time in production; recalibrate periodically as customer behaviour and churn drivers evolve.