Ensemble Methods Cheat Sheet

How bagging, boosting, and stacking combine multiple models to improve accuracy and robustness, with implementations using scikit-learn and XGBoost.

2 PagesIntermediateMar 10, 2026

Bagging & Random Forest

Parallel ensembles trained on bootstrap samples.

python

from sklearn.ensemble import BaggingClassifier, RandomForestClassifierfrom sklearn.tree import DecisionTreeClassifier# Bagging: train many models on bootstrap samples, average predictionsbagging = BaggingClassifier(    estimator=DecisionTreeClassifier(),    n_estimators=100,    max_samples=0.8,    bootstrap=True,    n_jobs=-1,    random_state=42,)bagging.fit(X_train, y_train)# Random Forest: bagging + random feature subsets at each splitrf = RandomForestClassifier(    n_estimators=200, max_depth=None, max_features="sqrt",    n_jobs=-1, random_state=42,)rf.fit(X_train, y_train)print(rf.feature_importances_)

Boosting & Stacking

Sequential ensembles and meta-learning.

python

from sklearn.ensemble import GradientBoostingClassifier, StackingClassifierfrom xgboost import XGBClassifierfrom sklearn.linear_model import LogisticRegression# Gradient Boosting: sequentially fit models to correct prior errorsgb = GradientBoostingClassifier(n_estimators=200, learning_rate=0.05, max_depth=3)gb.fit(X_train, y_train)# XGBoost: optimized, regularized gradient boostingxgb = XGBClassifier(n_estimators=300, learning_rate=0.05, max_depth=4,                     subsample=0.8, colsample_bytree=0.8, eval_metric="logloss")xgb.fit(X_train, y_train)# Stacking: combine predictions of base models via a meta-learnerstack = StackingClassifier(    estimators=[("rf", RandomForestClassifier()), ("xgb", xgb)],    final_estimator=LogisticRegression(),    cv=5,)stack.fit(X_train, y_train)

Ensemble Concepts

How different ensembling strategies work.

Bagging- trains base learners in parallel on bootstrap samples; reduces variance
Boosting- trains base learners sequentially, each correcting the previous one's errors; reduces bias
Random Forest- bagged decision trees with random feature subsampling at each split
Gradient Boosting- fits new trees to the residual/gradient of the loss from prior trees
XGBoost/LightGBM/CatBoost- optimized, regularized gradient boosting implementations
Stacking- trains a meta-model on the out-of-fold predictions of several base models
Voting- combines predictions via majority vote (hard) or averaged probabilities (soft)
Bias-variance tradeoff- bagging primarily reduces variance, boosting primarily reduces bias

Tuning Tips per Method

Practical guidance for common ensemble hyperparameters.

Random Forest n_estimators- more trees generally helps until diminishing returns; rarely overfits by adding more
Boosting learning_rate- lower learning rate + more estimators usually generalizes better, at the cost of training time
max_depth in boosting- shallow trees (3-8) are typical; deep trees in boosting overfit quickly
subsample/colsample_bytree- row and column subsampling adds regularization and reduces overfitting
Early stopping- monitor a validation set and stop boosting rounds once performance plateaus

Pro Tip

When stacking or blending models, always generate the meta-features using out-of-fold predictions (as StackingClassifier's cv parameter does) rather than predictions from models fit on the full training set — otherwise the meta-learner overfits to the base models' training performance.

Was this cheat sheet helpful?

Dimensionality Reduction Cheat Sheet

Gradient Descent Cheat Sheet

Explore Topics

#EnsembleMethods#EnsembleMethodsCheatSheet#DataScience#Intermediate#BaggingRandomForest#BoostingStacking#EnsembleConcepts#Tuning#Functions#MachineLearning#CheatSheet#SkillVeris

Sri Hayavadhana Info-Tech

Professional Web Designing Services