Gradient Boosting Cheat Sheet
A reference for gradient boosting covering XGBoost, LightGBM, and scikit-learn implementations, plus learning rate tuning, early stopping, and regularization.
2 PagesIntermediateFeb 28, 2026
XGBoost
Train with early stopping on a validation set.
python
import xgboost as xgbmodel = xgb.XGBClassifier( n_estimators=500, learning_rate=0.05, max_depth=4, subsample=0.8, colsample_bytree=0.8, eval_metric='logloss', early_stopping_rounds=20)model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
LightGBM
Train using LightGBM's native Dataset API.
python
import lightgbm as lgbtrain_data = lgb.Dataset(X_train, label=y_train)val_data = lgb.Dataset(X_val, label=y_val, reference=train_data)params = {'objective': 'binary', 'metric': 'auc', 'num_leaves': 31, 'learning_rate': 0.05}model = lgb.train( params, train_data, num_boost_round=1000, valid_sets=[val_data], callbacks=[lgb.early_stopping(stopping_rounds=50)])
scikit-learn GradientBoosting
Built-in gradient boosting without extra dependencies.
python
from sklearn.ensemble import GradientBoostingClassifiergb = GradientBoostingClassifier( n_estimators=200, learning_rate=0.1, max_depth=3, subsample=0.9)gb.fit(X_train, y_train)
Key Concepts
Core theory behind gradient boosting.
- Boosting- Sequentially fits trees to correct the residual errors of previous trees, unlike bagging's parallel trees
- Learning rate (shrinkage)- Scales each tree's contribution; lower values need more trees but usually generalize better
- Early stopping- Halts training once a validation metric stops improving, preventing overfitting
- Regularization (XGBoost)- gamma, reg_alpha (L1), and reg_lambda (L2) penalize tree complexity directly in the loss function
- Row/column subsampling- subsample and colsample_bytree add bagging-style randomness to each boosting round
SHAP Values for Tree Explainability
Get exact, fast Shapley attributions for any tree-boosting model via the tree-specific algorithm.
python
import shapexplainer = shap.TreeExplainer(model) # works for XGBoost, LightGBM, CatBoost, sklearn GBMshap_values = explainer.shap_values(X_test)# Global importance: mean absolute SHAP value per featureshap.summary_plot(shap_values, X_test, feature_names=feature_names, plot_type='bar')# Local explanation for a single predictionshap.force_plot(explainer.expected_value, shap_values[0], X_test.iloc[0])
Monotonic Constraints
Force predictions to move consistently with a feature when domain knowledge demands it.
python
import xgboost as xgb# +1 = prediction must be non-decreasing in this feature# -1 = prediction must be non-increasing# 0 = no constraint# Order must match the column order of X_trainmonotone = (1, -1, 0, 0, 1)model = xgb.XGBClassifier( n_estimators=300, max_depth=4, learning_rate=0.05, monotone_constraints=monotone,)model.fit(X_train, y_train)# Useful for credit/risk models where e.g. 'income' must never *decrease*# approval probability, regardless of what noisy data suggests.
Native Categorical Handling in LightGBM
Skip one-hot encoding entirely by letting LightGBM split on categories directly.
python
import lightgbm as lgbimport pandas as pdX_train['city'] = X_train['city'].astype('category')X_val['city'] = X_val['city'].astype('category')train_data = lgb.Dataset(X_train, label=y_train, categorical_feature=['city'])val_data = lgb.Dataset(X_val, label=y_val, reference=train_data)params = { 'objective': 'binary', 'metric': 'auc', 'max_bin': 255, # histogram resolution per feature 'min_data_per_group': 50, # regularizes rare categories}model = lgb.train(params, train_data, valid_sets=[val_data], callbacks=[lgb.early_stopping(50)])# LightGBM finds optimal category groupings via Fisher partitioning,# which beats naive one-hot for high-cardinality columns.
Custom & Quantile Objectives
Optimize beyond mean-squared error, e.g. for prediction intervals or asymmetric costs.
python
import xgboost as xgb# Built-in quantile regression (XGBoost >= 2.0)model_q10 = xgb.XGBRegressor(objective='reg:quantileerror', quantile_alpha=0.1)model_q90 = xgb.XGBRegressor(objective='reg:quantileerror', quantile_alpha=0.9)model_q10.fit(X_train, y_train)model_q90.fit(X_train, y_train)# [model_q10.predict(X_test), model_q90.predict(X_test)] gives an 80% interval# Fully custom objective: gradient + hessian for asymmetric squared errordef asymmetric_mse(y_true, y_pred): residual = y_pred - y_true grad = np.where(residual > 0, 2 * 3 * residual, 2 * residual) hess = np.where(residual > 0, 2 * 3, 2) return grad, hessbooster = xgb.train({'max_depth': 4}, xgb.DMatrix(X_train, y_train), num_boost_round=200, obj=asymmetric_mse)
Boosting Internals
What actually happens under the hood between XGBoost, LightGBM, and sklearn.
- Newton boosting (2nd-order)- XGBoost and LightGBM fit each tree using both gradient and Hessian of the loss, giving faster, more precise convergence than sklearn's gradient-only GBM
- Histogram binning- Continuous features are bucketed into a fixed number of bins (max_bin) before split-finding, trading a little precision for large speedups on big data
- Leaf-wise vs level-wise growth- LightGBM grows the single best leaf next (leaf-wise), reaching lower loss faster but risking overfitting on small data; XGBoost's default grows level-by-level
- DART booster- Applies dropout to trees during training (randomly muting some previous trees per round) to combat over-specialization in long boosting runs
- tree_method='hist'/'gpu_hist'- Histogram-based (and GPU) tree construction is now the default in XGBoost, replacing the exact greedy method for large datasets
- Second-order regularization (gamma, lambda)- gamma sets a minimum loss reduction to justify a split; reg_lambda (L2) shrinks leaf weights, both acting directly inside the split-gain formula
- Feature bundling (EFB)- LightGBM's Exclusive Feature Bundling merges mutually-exclusive sparse features into one, cutting effective dimensionality for one-hot-heavy data
Pro Tip
Tune learning_rate and n_estimators together — halve the learning rate and roughly double n_estimators, using early stopping on a held-out validation set, to trade extra compute for better generalization.
Was this cheat sheet helpful?
Explore Topics
#GradientBoosting#GradientBoostingCheatSheet#DataScience#Intermediate#XGBoost#LightGBM#ScikitLearnGradientBoosting#KeyConcepts#MachineLearning#CheatSheet#SkillVeris
