Bagging — bootstrap aggregating — is the ensemble technique that reduces model variance by training multiple copies of the same algorithm on different bootstrap samples of the training data and averaging their predictions. It converts any high-variance, low-bias learner (such as a deep decision tree) into a stable ensemble by exploiting a fundamental statistical property: the average of multiple correlated predictions has lower variance than any single prediction, and the more independent those predictions are, the more variance is eliminated.
Understanding bagging deeply matters beyond random forests because it is the foundational principle underlying most ensemble methods. Gradient boosting explicitly builds on bagging's limitations, stacking learns how to combine predictions that bagging averages naively, and variance reduction through ensemble averaging is the mathematical core of why any ensemble outperforms its members. This lesson covers the mechanics of bootstrap sampling, the variance-reduction mathematics, the out-of-bag estimator, and the conditions under which bagging helps versus where it fails.