Bagging
Bagging, short for bootstrap aggregating, is an ensemble learning method that trains multiple copies of a model on different randomly resampled (bootstrapped) versions of the training data and averages or votes their predictions to reduce…
Definition
Bagging, short for bootstrap aggregating, is an ensemble learning method that trains multiple copies of a model on different randomly resampled (bootstrapped) versions of the training data and averages or votes their predictions to reduce variance and improve generalization.
Overview
Bagging addresses a common weakness of flexible, high-capacity models: they can be highly sensitive to the specific training data they see, producing very different outputs if the training set changes slightly, a property known as high variance. Bagging exploits the law of large numbers by generating many slightly different versions of the training set through bootstrap sampling (sampling with replacement) and training an independent model on each version, then aggregating their predictions to cancel out much of this variance. The process runs in three steps: first, generate a set number of bootstrap samples from the original training data, each the same size as the original but containing repeated and missing examples due to sampling with replacement; second, train a separate base model, often of the same type and hyperparameters, on each bootstrap sample independently; third, combine the predictions from all trained models — typically by averaging predicted values for regression tasks or by majority vote for classification tasks. Because each model sees a slightly different dataset, they make somewhat different errors, and averaging or voting cancels out much of this uncorrelated error while preserving the shared signal that all models learned correctly. This makes bagging especially effective with base learners that are individually accurate but unstable, such as deep decision trees, which is why bagged decision trees form the foundation of the random forest algorithm. Bagging is contrasted with boosting, another major ensembling family, which builds models sequentially, with each new model focused on correcting the errors of the previous ensemble, rather than training all models independently and in parallel. Bagging generally trades a modest amount of interpretability and training simplicity for meaningfully improved robustness and accuracy, and it remains one of the most widely taught and applied ensemble techniques in practical machine learning.
Key Concepts
- Generates multiple bootstrap-resampled training sets with replacement
- Trains an independent base model on each resampled dataset
- Aggregates predictions via averaging or majority vote
- Primarily reduces prediction variance rather than bias
- Base models train independently and in parallel
- Forms the foundation of the random forest algorithm
- Most beneficial for high-variance base learners like decision trees
- Contrasts with boosting, which trains models sequentially