Ensemble Methods Cheat Sheet
How bagging, boosting, and stacking combine multiple models to improve accuracy and robustness, with implementations using scikit-learn and XGBoost.
2 PagesIntermediateMar 10, 2026
Bagging & Random Forest
Parallel ensembles trained on bootstrap samples.
python
from sklearn.ensemble import BaggingClassifier, RandomForestClassifierfrom sklearn.tree import DecisionTreeClassifier# Bagging: train many models on bootstrap samples, average predictionsbagging = BaggingClassifier( estimator=DecisionTreeClassifier(), n_estimators=100, max_samples=0.8, bootstrap=True, n_jobs=-1, random_state=42,)bagging.fit(X_train, y_train)# Random Forest: bagging + random feature subsets at each splitrf = RandomForestClassifier( n_estimators=200, max_depth=None, max_features="sqrt", n_jobs=-1, random_state=42,)rf.fit(X_train, y_train)print(rf.feature_importances_)
Boosting & Stacking
Sequential ensembles and meta-learning.
python
from sklearn.ensemble import GradientBoostingClassifier, StackingClassifierfrom xgboost import XGBClassifierfrom sklearn.linear_model import LogisticRegression# Gradient Boosting: sequentially fit models to correct prior errorsgb = GradientBoostingClassifier(n_estimators=200, learning_rate=0.05, max_depth=3)gb.fit(X_train, y_train)# XGBoost: optimized, regularized gradient boostingxgb = XGBClassifier(n_estimators=300, learning_rate=0.05, max_depth=4, subsample=0.8, colsample_bytree=0.8, eval_metric="logloss")xgb.fit(X_train, y_train)# Stacking: combine predictions of base models via a meta-learnerstack = StackingClassifier( estimators=[("rf", RandomForestClassifier()), ("xgb", xgb)], final_estimator=LogisticRegression(), cv=5,)stack.fit(X_train, y_train)
Ensemble Concepts
How different ensembling strategies work.
- Bagging- trains base learners in parallel on bootstrap samples; reduces variance
- Boosting- trains base learners sequentially, each correcting the previous one's errors; reduces bias
- Random Forest- bagged decision trees with random feature subsampling at each split
- Gradient Boosting- fits new trees to the residual/gradient of the loss from prior trees
- XGBoost/LightGBM/CatBoost- optimized, regularized gradient boosting implementations
- Stacking- trains a meta-model on the out-of-fold predictions of several base models
- Voting- combines predictions via majority vote (hard) or averaged probabilities (soft)
- Bias-variance tradeoff- bagging primarily reduces variance, boosting primarily reduces bias
Tuning Tips per Method
Practical guidance for common ensemble hyperparameters.
- Random Forest n_estimators- more trees generally helps until diminishing returns; rarely overfits by adding more
- Boosting learning_rate- lower learning rate + more estimators usually generalizes better, at the cost of training time
- max_depth in boosting- shallow trees (3-8) are typical; deep trees in boosting overfit quickly
- subsample/colsample_bytree- row and column subsampling adds regularization and reduces overfitting
- Early stopping- monitor a validation set and stop boosting rounds once performance plateaus
Extra Trees & Histogram-Based Boosting
Faster ensemble variants that trade a bit of per-tree quality for speed at scale.
python
from sklearn.ensemble import ExtraTreesClassifier, HistGradientBoostingClassifier# Extremely Randomized Trees: like RF, but split thresholds are also randomized# (not just the feature subset) -> lower variance, faster to train, slightly higher biaset = ExtraTreesClassifier( n_estimators=300, max_features="sqrt", n_jobs=-1, random_state=42,)et.fit(X_train, y_train)# Histogram-based GBM: bins continuous features into ~256 buckets before# splitting -> O(n_bins) split search instead of O(n_samples), scales to# millions of rows; native NaN support, no imputation neededhgb = HistGradientBoostingClassifier( max_iter=300, learning_rate=0.05, max_leaf_nodes=31, l2_regularization=1.0, early_stopping=True, validation_fraction=0.1, n_iter_no_change=20, random_state=42,)hgb.fit(X_train, y_train)
LightGBM & CatBoost Native Categoricals
Avoid manual one-hot/target encoding by letting the booster handle categorical splits directly.
python
import lightgbm as lgbfrom catboost import CatBoostClassifier# LightGBM: pass categorical columns explicitly, it uses Fisher-optimal# partitioning instead of naive one-hot encodingtrain_set = lgb.Dataset(X_train, label=y_train, categorical_feature=["city", "device"])params = { "objective": "binary", "metric": "auc", "num_leaves": 31, "learning_rate": 0.03, "feature_fraction": 0.8, "bagging_fraction": 0.8, "bagging_freq": 5,}booster = lgb.train( params, train_set, num_boost_round=1000, valid_sets=[lgb.Dataset(X_val, label=y_val)], callbacks=[lgb.early_stopping(50)],)# CatBoost: ordered target statistics avoid target leakage from naive# mean-encoding of high-cardinality categoricalscb = CatBoostClassifier( iterations=1000, learning_rate=0.03, depth=6, cat_features=["city", "device"], verbose=False, early_stopping_rounds=50,)cb.fit(X_train, y_train, eval_set=(X_val, y_val))
Permutation Importance & SHAP
Impurity-based importances are biased toward high-cardinality features; use model-agnostic alternatives instead.
python
from sklearn.inspection import permutation_importanceimport shap# Permutation importance: shuffle one feature at a time on held-out data and# measure the drop in score -> unbiased by cardinality, works for any estimatorresult = permutation_importance( rf, X_test, y_test, n_repeats=10, random_state=42, n_jobs=-1,)for i in result.importances_mean.argsort()[::-1][:10]: print(f"{X_test.columns[i]}: {result.importances_mean[i]:.4f} +/- {result.importances_std[i]:.4f}")# SHAP TreeExplainer: exact Shapley values for tree ensembles in polynomial# time, giving per-prediction, signed, additive attributionsexplainer = shap.TreeExplainer(xgb)shap_values = explainer.shap_values(X_test)shap.summary_plot(shap_values, X_test) # global feature impactshap.force_plot(explainer.expected_value, shap_values[0], X_test.iloc[0]) # single prediction
Monotonic Constraints & Imbalanced Ensembles
Encode domain knowledge as constraints and correct for class imbalance inside the ensemble instead of naive resampling.
python
from xgboost import XGBClassifierfrom imblearn.ensemble import BalancedRandomForestClassifier, EasyEnsembleClassifier# Monotonic constraints: force prediction to be non-decreasing (1) or# non-increasing (-1) in a given feature, e.g. credit_score should never# hurt approval odds as it increasesxgb_mono = XGBClassifier( n_estimators=300, max_depth=4, monotone_constraints=(1, 0, -1), # one entry per feature, in column order)xgb_mono.fit(X_train, y_train)# BalancedRandomForest: each tree is grown on a class-balanced bootstrap# sample instead of the raw imbalanced data -> better recall on the minority classbrf = BalancedRandomForestClassifier(n_estimators=300, sampling_strategy="auto", random_state=42)brf.fit(X_train, y_train)# EasyEnsemble: bags multiple AdaBoost learners, each trained on an# under-sampled balanced subset, then averages -> robust to severe imbalanceeec = EasyEnsembleClassifier(n_estimators=10, random_state=42)eec.fit(X_train, y_train)
Advanced Pitfalls & Diagnostics
Failure modes that only show up once you push ensembles into production.
- OOB score- for bagging/RF, set oob_score=True to get a near-free validation estimate from samples excluded by each tree's bootstrap, without a separate holdout
- Probability miscalibration- tree ensembles (especially boosted ones) often produce over/under-confident probabilities; wrap with CalibratedClassifierCV (isotonic or sigmoid) before using scores downstream
- Extrapolation blindness- tree-based ensembles cannot extrapolate beyond the range of training feature values; a linear model or explicit feature engineering is needed for out-of-range inputs
- Leakage via target encoding- mean/target-encoded categoricals fed into boosting must use out-of-fold encoding, or the ensemble will memorize the training target through the encoding
- Stacking meta-feature leakage- generate base-model predictions for the meta-learner using cross_val_predict/out-of-fold folds, never predictions from models fit on the full training set
- Correlated base learners- ensembling near-identical models (e.g. 5 XGBoost runs with different seeds only) yields little variance reduction; diversify algorithm family, features, or preprocessing
- GOSS / EFB (LightGBM)- Gradient-based One-Side Sampling keeps high-gradient samples and subsamples low-gradient ones; Exclusive Feature Bundling merges sparse mutually-exclusive features to speed up histogram building
Pro Tip
When stacking or blending models, always generate the meta-features using out-of-fold predictions (as StackingClassifier's cv parameter does) rather than predictions from models fit on the full training set — otherwise the meta-learner overfits to the base models' training performance.
Was this cheat sheet helpful?
Explore Topics
#EnsembleMethods#EnsembleMethodsCheatSheet#DataScience#Intermediate#BaggingRandomForest#BoostingStacking#EnsembleConcepts#Tuning#Functions#MachineLearning#CheatSheet#SkillVeris
