Boosting
Boosting is an ensemble learning technique that builds a sequence of models, where each new model is trained to correct the errors made by the combined ensemble of previous models, ultimately combining them into a single, typically…
Definition
Boosting is an ensemble learning technique that builds a sequence of models, where each new model is trained to correct the errors made by the combined ensemble of previous models, ultimately combining them into a single, typically weighted, strong predictor.
Overview
Unlike bagging, where models are trained independently in parallel on resampled data, boosting trains models sequentially and adaptively. Each new base model, or weak learner (often a shallow decision tree), is trained specifically to focus on the examples that previous models in the ensemble got wrong, either by reweighting misclassified training examples so they receive more attention, as in the classic AdaBoost algorithm, or by directly fitting the new model to the residual errors of the current ensemble, as in gradient boosting. Gradient boosting, the more widely used modern approach, frames the whole ensemble-building process as gradient descent in function space: at each iteration, a new weak learner is fit to approximate the negative gradient of the loss function with respect to the current ensemble's predictions, effectively learning to correct the ensemble's current mistakes in the direction that most reduces the loss. The predictions of all weak learners are then combined, typically with learning-rate-scaled weights, to form the final strong model. Popular, highly optimized implementations of gradient boosting include XGBoost, LightGBM, and CatBoost, which dominate many tabular data machine learning competitions and production systems. Boosting's key strength is its ability to reduce bias — turning a collection of individually weak, simple models into a highly accurate ensemble — but this sequential, error-correcting process also makes it more prone to overfitting than bagging if not carefully regularized, since later models can start fitting noise in the residual errors. Common regularization strategies include limiting tree depth, using a small learning rate, subsampling data or features per iteration, and early stopping based on validation performance. Because each model in a boosted ensemble depends on the state of all previous models, boosting cannot be parallelized across the sequence of iterations in the way bagging can, though individual tree construction within each iteration can still be parallelized.
Key Concepts
- Trains models sequentially, each correcting the errors of the previous ensemble
- Primarily reduces bias, converting weak learners into a strong ensemble
- Gradient boosting fits each new model to the ensemble's residual errors
- Popular implementations include XGBoost, LightGBM, and CatBoost
- More prone to overfitting than bagging without careful regularization
- Regularized via learning rate, tree depth limits, and subsampling
- Cannot be parallelized across sequential iterations like bagging can
- Dominant technique for structured/tabular data competitions and applications