Gradient Boosting
Gradient Boosting is an ensemble learning technique that builds a model as a sequence of weak learners, typically shallow decision trees, where each new learner is trained to correct the residual errors of the combined ensemble so far.
Definition
Gradient Boosting is an ensemble learning technique that builds a model as a sequence of weak learners, typically shallow decision trees, where each new learner is trained to correct the residual errors of the combined ensemble so far.
Overview
Gradient Boosting frames model building as a numerical optimization problem: at each iteration, it fits a new weak learner (usually a shallow decision tree) to the negative gradient of a loss function with respect to the current ensemble's predictions — effectively, the direction in which predictions need to move to reduce error the most. The new tree's output is then added to the ensemble, scaled by a learning rate that controls how aggressively each step updates the overall model, and the process repeats for a fixed number of boosting rounds or until validation performance stops improving. This sequential, error-correcting structure differs fundamentally from bagging methods like Random Forest, which train trees independently and in parallel; gradient boosting trees are trained one after another, each dependent on the errors of all previous trees. This generally lets gradient boosting achieve lower bias and higher accuracy than bagging on many tasks, at the cost of being more prone to overfitting if not carefully regularized (via tree depth limits, learning rate, subsampling, and early stopping) and slower to train since trees cannot be built fully in parallel. Gradient boosting's loss-gradient framing makes it highly general — by swapping the loss function, the same algorithm handles regression, classification, ranking, and even custom objectives. Highly optimized implementations, most notably XGBoost, along with LightGBM and CatBoost, added engineering innovations like histogram-based split finding, regularization terms, and native handling of categorical features and missing values, making gradient boosting the dominant technique for structured/tabular data problems in industry and machine learning competitions, frequently outperforming both simpler tree ensembles and deep neural networks on that data type.
Key Concepts
- Sequentially builds an ensemble, each learner correcting prior residual errors
- Fits new learners to the negative gradient of a loss function (functional gradient descent)
- Learning rate controls how aggressively each new tree updates the ensemble
- Generalizes to regression, classification, ranking, and custom loss objectives
- Requires regularization (tree depth, subsampling, early stopping) to avoid overfitting
- Sequential training is slower and less parallelizable than bagging methods
- Basis for highly optimized libraries like XGBoost, LightGBM, and CatBoost
- State-of-the-art performance on many structured/tabular data benchmarks