Train/Test Split and Cross-Validation

Analogy categoryAnalogies

A model that is evaluated on the exact data it was trained on will almost always look better than it truly is, because it has had the opportunity to memorize quirks of that specific data rather than learn generalizable patterns. The train/test split addresses this by partitioning the dataset before training begins: one portion (commonly 70-80%) is used to fit the model, and a held-out portion (the remaining 20-30%) is used purely to measure how well the model generalizes to data it has never seen. The test set must never influence training decisions, including hyperparameter choices, or the estimate becomes optimistic again.

🏏

Cricket analogy: A batter's net-session form against bowling machines is not proof of match temperament; selectors hold back live match footage, unseen during practice, to judge whether the technique truly transfers.

Why a Single Split Isn't Enough

A single train/test split gives one performance number, but that number depends heavily on which rows happened to land in the test set — an unlucky split can make a good model look mediocre, and a lucky split can make a mediocre model look great. This is especially problematic with small datasets, where a handful of rows can swing the metric substantially. It also tempts practitioners into repeatedly tweaking hyperparameters based on test set performance, which quietly turns the test set into a second training set and destroys its validity as an unbiased estimate.

🏏

Cricket analogy: A single day-night match where rain reduces overs to 20 can make a strong team look weak or a weak team look strong; one game is a noisy sample of true ability.

K-Fold Cross-Validation

K-fold cross-validation splits the training data into k equally sized folds (typically k=5 or k=10). The model is trained k times, each time holding out a different fold as the validation set and training on the remaining k-1 folds. The final performance estimate is the average (and standard deviation) across all k runs, which uses every row for both training and validation while still keeping each validation fold unseen during that fold's training. This produces a far more stable, lower-variance estimate than a single split, and the standard deviation across folds also tells you how sensitive the model is to which specific data it sees. A separate, untouched test set is still held out entirely, reserved only for a final, one-time evaluation after all model and hyperparameter decisions are locked in using cross-validation on the training data.

🏏

Cricket analogy: Instead of judging a batter on one series, selectors watch him bat across 5 different conditions (pace, spin, swing, bounce, flat decks), averaging performance across all of them for a stable rating, then still reserve a brand-new tour to confirm.

python

from sklearn.model_selection import train_test_split, KFold, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

# Step 1: hold out a final test set, untouched until the very end
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Step 2: use k-fold CV on the training set to estimate generalization
model = LogisticRegression(max_iter=5000)
kf = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_train, y_train, cv=kf, scoring='accuracy')

print('per-fold accuracy:', scores.round(3))
print('mean +/- std:', scores.mean().round(3), '+/-', scores.std().round(3))

# Step 3: only after model/hyperparameters are finalized, evaluate once on X_test

Stratified k-fold cross-validation preserves the class proportions of the original dataset within every fold. For an imbalanced dataset (say, 5% positive class), plain random folds could by chance produce a fold with almost no positive examples, making that fold's validation score meaningless — stratification prevents this.

Never use the test set to choose between models or tune hyperparameters. If you find yourself running an experiment, checking test accuracy, and adjusting a hyperparameter based on that result, you are effectively training on the test set by proxy, and your final reported number will be optimistically biased.

A train/test split reserves data the model never sees during training, to estimate real-world generalization.
A single split gives a noisy, high-variance estimate, especially on small datasets.
K-fold cross-validation trains/validates k times over rotating folds and averages the results for a stable estimate.
Stratified k-fold preserves class proportions in each fold, which matters for imbalanced classification.
The test set should remain completely untouched until final evaluation, after all tuning is done via cross-validation.
Repeatedly checking test performance during tuning silently turns the test set into a training signal.

Practice what you learned

1. What is the main problem with using only a single train/test split to evaluate a model?

2. In k-fold cross-validation with k=5, how many times is the model trained?

3. Why is stratified k-fold preferred for imbalanced classification datasets?

4. What happens if you repeatedly tune hyperparameters based on test set performance?

5. What does the standard deviation across k-fold scores tell you?

Was this page helpful?

Feature Scaling and Normalization

Topics covered

#Python#MachineLearningBasicsStudyNotes#MachineLearning#TrainTestSplitAndCrossValidation#Train#Test#Split#Cross#Testing#StudyNotes#SkillVeris

Train/Test Split and Cross-Validation