Random Forests and Ensembles
Ensemble methods combine predictions from multiple models to produce a result that is typically more accurate and more stable than any individual model alone. The intuition mirrors 'wisdom of the crowd': if individual models make errors that are at least somewhat independent of one another, averaging or voting across them cancels out much of that error while preserving the shared signal. Two major families dominate practice — bagging, which trains many models in parallel on different random subsets of data and averages their outputs (random forests being the canonical example), and boosting, which trains models sequentially, each one focusing on correcting the errors of its predecessors (gradient boosting and AdaBoost being canonical examples).
Cricket analogy: A five-man selection panel's combined vote on team selection is usually better than any single selector's judgment alone, since their individual blind spots cancel out; bagging is like polling many panels on random subsets of players' stats, while boosting is like a chief selector correcting the previous panel's specific misses one round at a time.
Random Forests: Bagging Plus Feature Randomness
A random forest builds many decision trees, each trained on a bootstrap sample (a random sample drawn with replacement, the same size as the original training set) of the training data — this is the 'bagging' (bootstrap aggregating) part. On top of that, at each split within each tree, only a random subset of features is considered as candidates, rather than all features — this extra randomness decorrelates the trees further, since without it, trees would tend to make the same strong early splits on the most predictive features and end up highly correlated with each other, which would limit the variance-reduction benefit of averaging. Final predictions are made by averaging (regression) or majority voting (classification) across all trees in the forest.
Cricket analogy: Random forests are like assembling many playing XIs each drawn from a randomly resampled squad list (bootstrap sample), and at each team-selection meeting only a random subset of criteria (say, only bowling stats, not batting) is considered, so no two XIs get built the same way.
Why Averaging Reduces Variance
A single decision tree is a high-variance, low-bias model — it can fit training data very closely but is unstable across different training samples. Averaging predictions from many such trees, each fit on a slightly different bootstrap sample, keeps the low bias (each tree can still fit complex patterns) while substantially reducing variance, since the trees' individual errors partially cancel out when averaged, provided those errors are not perfectly correlated. This is why random forests are far more robust to overfitting than a single unconstrained decision tree, even though individual trees in the forest are often grown deep and largely unpruned.
Cricket analogy: A single all-rounder's form is wildly inconsistent match to match (high variance) even though he's technically gifted (low bias); averaging the performances of many such all-rounders picked slightly differently each match keeps that skill while smoothing out the streaky form.
import numpy as np
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=600, n_features=10, n_informative=6, random_state=6)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=6)
models = {
"single tree": DecisionTreeClassifier(random_state=6),
"random forest": RandomForestClassifier(n_estimators=200, max_features="sqrt", random_state=6),
"gradient boosting": GradientBoostingClassifier(n_estimators=200, learning_rate=0.05, random_state=6),
}
for name, model in models.items():
cv_scores = cross_val_score(model, X_train, y_train, cv=5)
model.fit(X_train, y_train)
test_acc = model.score(X_test, y_test)
print(f"{name:>18}: cv acc={cv_scores.mean():.3f} test acc={test_acc:.3f}")
# Feature importances from the random forest
rf = models["random forest"]
importances = rf.feature_importances_
top_features = np.argsort(importances)[::-1][:3]
print("top 3 features by importance:", top_features, importances[top_features])Random forests provide a built-in, low-cost estimate of feature importance (based on average impurity reduction each feature contributes across all trees), and even an internal validation-like estimate called out-of-bag (OOB) error — since each tree is trained on roughly two-thirds of the data (due to bootstrap sampling with replacement), the remaining 'out-of-bag' third can be used to evaluate that tree without a separate held-out validation set.
Boosting and bagging solve different problems and should not be conflated: bagging (random forests) primarily reduces variance and works well with high-variance, low-bias base learners like deep trees, while boosting primarily reduces bias by sequentially correcting errors, and is more prone to overfitting if the number of boosting rounds or learning rate is not carefully controlled.
Gradient Boosting in Brief
Gradient boosting builds trees sequentially, where each new (typically shallow) tree is trained to predict the residual errors of the ensemble built so far, scaled by a learning rate that controls how much each new tree contributes. This sequential error-correction can achieve very strong predictive accuracy, often outperforming random forests on structured/tabular data, but it is more sensitive to hyperparameters (number of estimators, learning rate, tree depth) and more prone to overfitting if not tuned carefully — unlike random forests, adding more boosting rounds does not always improve generalization and can eventually hurt it.
Cricket analogy: Gradient boosting is like a batting coach reviewing each net session and having the next session focus specifically on correcting yesterday's dismissal pattern, with each correction scaled by a small learning rate; push too many correction rounds and the batter starts overfitting to quirks of the bowling machine.
- Ensembles combine multiple models to reduce error via averaging/voting, exploiting the idea that independent errors partially cancel out.
- Bagging (e.g., random forests) trains models in parallel on bootstrap samples and averages results, primarily reducing variance.
- Random forests add per-split feature randomness on top of bagging to decorrelate trees further.
- Boosting (e.g., gradient boosting) trains models sequentially, each correcting the errors of prior models, primarily reducing bias.
- Random forests provide built-in feature importances and out-of-bag error estimates.
- Boosting is more accuracy-prone but also more overfitting-prone than bagging if hyperparameters aren't tuned carefully.
Practice what you learned
1. What is the core mechanism that random forests use to decorrelate individual trees beyond simple bootstrap sampling?
2. Bagging methods like random forests primarily aim to reduce which source of error?
3. What is an out-of-bag (OOB) sample in the context of random forests?
4. How does gradient boosting fundamentally differ from random forests in how trees are built?
5. Why is gradient boosting generally considered more prone to overfitting than random forests if hyperparameters are not carefully tuned?
Was this page helpful?
You May Also Like
Decision Trees
Covers how decision trees split data recursively using impurity measures like Gini and entropy to produce interpretable, rule-based predictions.
Overfitting and Underfitting
The two failure modes of model fitting — memorizing noise in training data versus failing to capture real patterns — and the diagnostic and mitigation techniques for each.
Bias-Variance Tradeoff
A foundational concept explaining how model error decomposes into bias, variance, and irreducible noise, and how balancing them guides model complexity choices.
Hyperparameter Tuning
Explore systematic strategies — grid search, random search, and cross-validation-based tuning — for choosing model settings that are not learned directly from data.