Overfitting

Definition

Overfitting occurs when a machine learning model learns the training data too closely, including its noise and idiosyncrasies, resulting in poor performance on new, unseen data.

Overview

An overfit model essentially memorizes its training examples rather than learning generalizable patterns. It may achieve near-perfect accuracy on the data it was trained on, but perform noticeably worse on held-out validation or test data, because it has captured spurious correlations and noise specific to the training set rather than the underlying signal. Overfitting typically becomes more likely as model complexity increases relative to the amount of available training data — a model with many parameters, such as a deep neural network trained via gradient descent and backpropagation, has enough capacity to fit noise if not properly constrained. Common mitigation techniques include gathering more training data, applying regularization, using dropout, simplifying the model architecture, and monitoring performance on a separate validation set to stop training before performance on unseen data starts to degrade. Overfitting sits at one end of the bias-variance tradeoff, with underfitting at the other; a well-generalizing model needs to balance fitting the training data closely enough to be useful, without fitting it so closely that it fails to generalize. Techniques like cross-validation are commonly used to detect and guard against overfitting during model development.

Key Concepts

• Model performs well on training data but poorly on unseen data
• Often caused by excessive model complexity relative to data size
• Results in the model memorizing noise rather than general patterns
• Detected by comparing training performance to validation performance
• Mitigated with more data, regularization, dropout, or simpler models
• Sits opposite underfitting on the bias-variance tradeoff spectrum

Use Cases

Frequently Asked Questions

Related in AI Fundamentals