Cross-Validation Cheat Sheet
Methods for reliably estimating model generalization performance, covering k-fold, stratified, time-series, and leave-one-out cross-validation with scikit-learn.
2 PagesIntermediateMar 20, 2026
K-Fold Cross-Validation
Standard k-fold split and scoring.
python
from sklearn.model_selection import KFold, cross_val_scorefrom sklearn.ensemble import RandomForestClassifiermodel = RandomForestClassifier(random_state=42)kf = KFold(n_splits=5, shuffle=True, random_state=42)scores = cross_val_score(model, X, y, cv=kf, scoring="accuracy")print(f"Mean accuracy: {scores.mean():.3f} +/- {scores.std():.3f}")
Stratified & Time Series Splits
Preserve class balance or chronological order.
python
from sklearn.model_selection import StratifiedKFold, TimeSeriesSplit# Preserves class proportions in each fold -- use for classificationskf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)for train_idx, val_idx in skf.split(X, y): X_train, X_val = X[train_idx], X[val_idx] y_train, y_val = y[train_idx], y[val_idx]# Respects temporal order -- never shuffle time series datatscv = TimeSeriesSplit(n_splits=5)for train_idx, val_idx in tscv.split(X): X_train, X_val = X[train_idx], X[val_idx]
CV Strategies
Which splitting strategy to use and when.
- K-Fold- splits data into k equal folds; each fold used once as validation
- Stratified K-Fold- preserves class distribution in each fold; use for imbalanced classification
- Leave-One-Out (LOO)- k = n; expensive but low bias, high variance
- Time Series Split- expanding-window splits that respect chronological order, no shuffling
- Group K-Fold- ensures samples from the same group (e.g. patient, user) stay in one fold
- Repeated K-Fold- repeats k-fold multiple times with different splits for a more stable estimate
- Nested CV- outer loop for performance estimation, inner loop for hyperparameter tuning; avoids optimistic bias
Common Pitfalls
Mistakes that invalidate a cross-validation estimate.
- Leakage before splitting- scaling/imputing on the full dataset before CV leaks test info into training
- Shuffling time series- destroys temporal order and lets the model see the future
- Ignoring groups- random k-fold on grouped data (e.g. multiple rows per patient) leaks identity across folds
- Tuning on the same folds- using CV both to tune hyperparameters and report final performance overstates accuracy
- Too few folds on small data- small k on small datasets gives high-variance, unreliable estimates
GroupKFold to Prevent Group Leakage
Keep every row from the same entity (patient, user, device) confined to a single fold.
python
from sklearn.model_selection import GroupKFold, cross_val_scorefrom sklearn.ensemble import RandomForestClassifiergroups = df["patient_id"].valuesgkf = GroupKFold(n_splits=5)model = RandomForestClassifier(random_state=42)scores = cross_val_score(model, X, y, groups=groups, cv=gkf, scoring="roc_auc")print(f"Mean AUC: {scores.mean():.3f} +/- {scores.std():.3f}")# Manual split when you need the fold indices directlyfor train_idx, val_idx in gkf.split(X, y, groups=groups): assert set(groups[train_idx]).isdisjoint(set(groups[val_idx]))
Nested Cross-Validation
Separate the hyperparameter-tuning loop from the performance-estimation loop to get an unbiased score.
python
from sklearn.model_selection import GridSearchCV, cross_val_score, KFoldfrom sklearn.svm import SVCparam_grid = {"C": [0.1, 1, 10], "gamma": [0.01, 0.1, 1]}inner_cv = KFold(n_splits=4, shuffle=True, random_state=1)outer_cv = KFold(n_splits=5, shuffle=True, random_state=2)# Inner loop selects hyperparameters, outer loop scores that selection processclf = GridSearchCV(SVC(), param_grid, cv=inner_cv, scoring="accuracy")nested_scores = cross_val_score(clf, X, y, cv=outer_cv, scoring="accuracy")print(f"Nested CV accuracy: {nested_scores.mean():.3f} +/- {nested_scores.std():.3f}")
cross_validate() with Multiple Metrics
Score several metrics in one pass and compare train vs. validation to spot overfitting.
python
from sklearn.model_selection import cross_validatefrom sklearn.ensemble import GradientBoostingClassifierscoring = ["accuracy", "roc_auc", "f1", "precision", "recall"]results = cross_validate( GradientBoostingClassifier(), X, y, cv=5, scoring=scoring, return_train_score=True, n_jobs=-1,)for metric in scoring: train_mean = results[f"train_{metric}"].mean() test_mean = results[f"test_{metric}"].mean() gap = train_mean - test_mean print(f"{metric}: train={train_mean:.3f} val={test_mean:.3f} gap={gap:.3f}")
Purged & Embargoed CV for Financial Time Series
Custom splitter that removes overlapping-label leakage between train and validation windows.
python
import numpy as npclass PurgedKFold: """K-fold that purges training samples whose label window overlaps the validation window, plus an embargo period after it, to stop leakage from overlapping label horizons in financial/event data.""" def __init__(self, n_splits=5, embargo=0.01): self.n_splits = n_splits self.embargo = embargo def split(self, X, label_end_times): n = len(X) indices = np.arange(n) fold_bounds = np.linspace(0, n, self.n_splits + 1).astype(int) embargo_len = int(n * self.embargo) for i in range(self.n_splits): val_start, val_end = fold_bounds[i], fold_bounds[i + 1] val_idx = indices[val_start:val_end] purge_mask = (label_end_times[:val_start] >= val_start) if val_start > 0 else np.array([], dtype=bool) train_idx = np.concatenate([ indices[:val_start][~purge_mask] if val_start > 0 else indices[:0], indices[min(val_end + embargo_len, n):], ]) yield train_idx, val_idx
Diagnosing High CV Score Variance
Signals to check when fold-to-fold scores swing widely and the estimate feels unstable.
- Small dataset / few folds- each fold's validation set is tiny, so scores are noisy; increase k or use repeated CV
- Class/label imbalance- rare classes land unevenly across folds unless using StratifiedKFold
- Unaccounted grouping- correlated samples split across folds inflate apparent variance and optimism
- Outlier-heavy folds- a handful of extreme rows landing in one fold can dominate that fold's metric
- Non-stationary data- distribution drift over time means folds from different periods aren't comparable
- Metric sensitivity- some metrics (e.g. AUC on tiny positive counts) are inherently high-variance; report confidence intervals
Pro Tip
When tuning hyperparameters, use nested cross-validation (or a separate held-out test set) — evaluating on the same folds you tuned on gives an optimistically biased performance estimate.
Was this cheat sheet helpful?
Explore Topics
#CrossValidation#CrossValidationCheatSheet#DataScience#Intermediate#KFoldCrossValidation#Stratified#Time#Series#Functions#MachineLearning#CheatSheet#SkillVeris
