Overfitting and Underfitting

Analogy categoryAnalogies

Overfitting occurs when a model learns the training data too well — including its noise and idiosyncrasies — so that it performs excellently on training data but poorly on new, unseen data. Underfitting is the opposite failure: the model is too simple to capture the underlying structure of the data at all, performing poorly on both training and test data. Both are failures of generalization, but they point in opposite directions and require opposite remedies, which is why correctly diagnosing which one you're facing is the first step before applying any fix.

🏏

Cricket analogy: A batsman who dominates net practice against friendly bowling but collapses against real match-day pace is overfitting; one who struggles even in the nets is underfitting - opposite problems requiring a coach to first correctly diagnose which is happening before choosing a fix.

Diagnosing the Problem

The standard diagnostic tool is comparing training error to validation error. If training error is low but validation error is high, the gap indicates overfitting — the model has effectively memorized the training set rather than learning generalizable patterns. If both training and validation error are high and similar, the model is underfitting — it lacks the capacity or hasn't been given the right features to capture the true relationship. Learning curves, which plot both errors as a function of training set size, make this diagnosis visual: an overfitting model shows a persistent gap between the two curves even as more data is added, while an underfitting model shows both curves converging to a high error plateau.

🏏

Cricket analogy: If a batter's net-practice average is excellent but real-match average is poor, that gap signals overfitting - he's memorized the bowling machine rather than learning to read deliveries; if both are mediocre, that's underfitting, visible on a form curve plotted across a season.

Remedies

For overfitting, effective remedies include gathering more training data, simplifying the model (fewer features, shallower trees, fewer parameters), adding regularization (L1/L2 penalties, dropout in neural networks), and using techniques like cross-validation and early stopping to detect the onset of overfitting during training. For underfitting, remedies move in the opposite direction: increasing model capacity (more features, deeper trees, more layers), engineering better features that actually capture the signal, reducing regularization strength, and training for longer if the model hasn't yet converged. The bias-variance tradeoff formalizes this tension: underfitting corresponds to high bias, overfitting corresponds to high variance, and the goal of model selection is finding the sweet spot that minimizes total generalization error.

🏏

Cricket analogy: For an overfit batter who's memorized one bowling machine, the fix is facing varied bowlers, simplifying technique, and stopping once form plateaus; for an underfit batter, the fix is advanced drills, better shot selection, and longer practice - the tradeoff is a sweet spot between rigid and erratic.

python

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import learning_curve
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=500, n_features=10, noise=15, random_state=0)

# A very deep tree is prone to overfitting; a depth-1 stump is prone to underfitting
for max_depth, label in [(1, 'underfit-prone'), (None, 'overfit-prone')]:
    model = DecisionTreeRegressor(max_depth=max_depth, random_state=0)
    train_sizes, train_scores, val_scores = learning_curve(
        model, X, y, cv=5, scoring='r2', train_sizes=np.linspace(0.1, 1.0, 5)
    )
    gap = train_scores.mean(axis=1)[-1] - val_scores.mean(axis=1)[-1]
    print(f'{label} (max_depth={max_depth}): final train R2={train_scores.mean(axis=1)[-1]:.3f}, '
          f'val R2={val_scores.mean(axis=1)[-1]:.3f}, gap={gap:.3f}')

A useful mental model: underfitting is like studying only the chapter titles before an exam — you haven't learned enough to answer even easy questions. Overfitting is like memorizing the exact wording of last year's practice exam — you'll ace that exact test but flounder on this year's differently-worded questions covering the same material.

A common mistake is tuning hyperparameters by watching test set performance directly, then concluding the model generalizes well. This 'leaks' the test set into the model selection process and produces an overly optimistic estimate — always use a separate validation set or cross-validation for tuning, and reserve the test set for a single final evaluation.

Overfitting: low training error, high validation error — the model memorized noise instead of learning patterns.
Underfitting: high error on both training and validation data — the model lacks capacity to capture the signal.
Learning curves (error vs. training set size) are a standard visual diagnostic for distinguishing the two.
Overfitting remedies: more data, regularization, simpler models, early stopping, cross-validation.
Underfitting remedies: more model capacity, better features, less regularization, longer training.
The bias-variance tradeoff formalizes the balance: underfitting is high bias, overfitting is high variance.

Practice what you learned

1. What is the hallmark sign of overfitting when comparing training and validation error?

2. What is the hallmark sign of underfitting?

3. Which of these is an effective remedy specifically for overfitting?

4. Underfitting corresponds to which side of the bias-variance tradeoff?

5. Why is it a mistake to tune hyperparameters by directly observing performance on the final test set?

Was this page helpful?

ROC Curves and AUC

Topics covered

#Python#MachineLearningBasicsStudyNotes#MachineLearning#OverfittingAndUnderfitting#Overfitting#Underfitting#Diagnosing#Problem#StudyNotes#SkillVeris

Overfitting and Underfitting