Gradient boosting is the most powerful and widely-used ensemble method for tabular data, consistently topping Kaggle leaderboards and industry benchmarks. Unlike bagging, which builds trees independently in parallel to reduce variance, gradient boosting builds trees sequentially, where each new tree corrects the errors of all previous trees. It is a bias-reduction technique: the ensemble starts with a high-bias predictor and iteratively adds models that specifically target the residual errors, progressively reducing bias while carefully controlling variance through regularisation and learning rates.
XGBoost and LightGBM are the two dominant gradient boosting libraries in production data science, both extending the original gradient boosting framework with hardware-level optimisations, regularisation, and handling of missing values that make them dramatically faster and more accurate than scikit-learn's GradientBoostingClassifier. Understanding gradient boosting's algorithmic core — gradient descent in function space, additive models, residual fitting — and the practical tuning of XGBoost and LightGBM is one of the highest-value skills in applied machine learning.