Bootstrap Aggregating

Definition

Bootstrap aggregating, commonly called bagging, is an ensemble learning technique that trains multiple instances of a model on different bootstrap-sampled subsets of the training data and combines their predictions, typically by averaging or majority voting, to reduce variance and improve robustness.

Overview

Bootstrap aggregating gets its name from combining two ideas: bootstrap sampling, a statistical resampling technique where new datasets are created by sampling with replacement from the original training data, and aggregating, the process of combining predictions from multiple models. In bagging, several bootstrap samples are drawn from the original training set, each the same size as the original but containing duplicates and omitting some original examples (typically around 63% of unique examples are included in each bootstrap sample). A separate model is trained independently on each bootstrap sample, and the final prediction is formed by averaging the outputs (for regression) or taking a majority vote (for classification) across all the individual models. The core benefit of bagging is variance reduction. Individual models, especially high-variance ones like deep decision trees, tend to overfit to the specific data they were trained on and can produce quite different predictions if trained on slightly different data samples. By training many such models on different bootstrap samples and averaging their outputs, bagging smooths out this variance without significantly increasing bias, generally producing a more stable and accurate ensemble than any single model. Bagging's most well-known application is the random forest algorithm, which combines bagging with an additional layer of randomness — at each split in each decision tree, only a random subset of features is considered — further decorrelating the individual trees and improving the ensemble's overall performance. Because each model in a bagging ensemble is trained independently, the process parallelizes naturally, unlike boosting, where models are trained sequentially and each depends on the errors of the previous one. Bagging works best with base models that have low bias but high variance, such as unpruned decision trees; it provides little benefit for models that are already low-variance, like linear regression, since there is little variance left to reduce.

Key Concepts

• Trains multiple models on different bootstrap-sampled subsets of the training data
• Combines predictions via averaging (regression) or majority voting (classification)
• Primarily reduces variance rather than bias
• Base models are trained independently, enabling natural parallelization
• Most well-known implementation is the random forest algorithm
• Works best with high-variance base models like unpruned decision trees
• Each bootstrap sample typically contains about 63% of unique training examples
• Out-of-bag samples can be used for an internal validation-like estimate

Use Cases

Frequently Asked Questions

Related in AI Fundamentals