Random Forest Cheat Sheet
A cheat sheet for Random Forest covering bagging, feature randomness, out-of-bag scoring, hyperparameter tuning, and feature importance in scikit-learn.
2 PagesIntermediateMar 2, 2026
Classifier with scikit-learn
Fit a forest and check its out-of-bag score.
python
from sklearn.ensemble import RandomForestClassifierrf = RandomForestClassifier( n_estimators=300, max_depth=None, max_features='sqrt', min_samples_leaf=2, oob_score=True, n_jobs=-1, random_state=42)rf.fit(X_train, y_train)print('OOB score:', rf.oob_score_) # validation-free accuracy estimateprint('Test accuracy:', rf.score(X_test, y_test))
Regressor & Feature Importance
Random Forest for regression tasks.
python
from sklearn.ensemble import RandomForestRegressorreg = RandomForestRegressor(n_estimators=200, n_jobs=-1, random_state=42)reg.fit(X_train, y_train)importances = reg.feature_importances_ # mean decrease in impurity per feature
Hyperparameter Search
Tune key forest hyperparameters.
python
from sklearn.model_selection import RandomizedSearchCVparam_dist = { 'n_estimators': [100, 300, 500], 'max_depth': [None, 10, 20, 30], 'min_samples_leaf': [1, 2, 4],}search = RandomizedSearchCV(RandomForestClassifier(), param_dist, cv=5, n_iter=20, n_jobs=-1)search.fit(X_train, y_train)print(search.best_params_)
Key Concepts
Core theory behind Random Forest.
- Bagging- Each tree trains on a bootstrap sample (random sample with replacement) of the training data
- Feature randomness- Each split considers only a random subset of features (max_features), decorrelating the trees
- Out-of-bag (OOB) score- Free validation estimate using the roughly 37% of samples excluded from each tree's bootstrap sample
- n_estimators- Number of trees in the forest; more trees reduce variance with diminishing returns on compute
- Feature importance- Mean decrease in impurity across all trees, or more robust permutation importance
Extremely Randomized Trees
Trade a bit of bias for lower variance by also randomizing split thresholds, not just features.
python
from sklearn.ensemble import ExtraTreesClassifier, RandomForestClassifierfrom sklearn.model_selection import cross_val_score# ExtraTrees picks split thresholds *randomly* per candidate feature# instead of searching for the optimal threshold (RandomForest's approach).# This decorrelates trees further and trains faster (no threshold search).et = ExtraTreesClassifier(n_estimators=300, max_features='sqrt', n_jobs=-1, random_state=42)rf = RandomForestClassifier(n_estimators=300, max_features='sqrt', n_jobs=-1, random_state=42)for name, model in [('ExtraTrees', et), ('RandomForest', rf)]: scores = cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc') print(f'{name}: {scores.mean():.4f} +/- {scores.std():.4f}')
Permutation Importance on Held-Out Data
Measure true predictive importance by shuffling each feature and watching score drop, not impurity.
python
from sklearn.inspection import permutation_importanceimport numpy as npresult = permutation_importance( rf, X_test, y_test, n_repeats=30, random_state=42, n_jobs=-1, scoring='roc_auc')order = result.importances_mean.argsort()[::-1]for i in order[:10]: print(f'{feature_names[i]:<25} {result.importances_mean[i]:.4f} +/- {result.importances_std[i]:.4f}')# Features whose CI crosses zero add nothing beyond noise -> candidates to dropunreliable = [feature_names[i] for i in order if result.importances_mean[i] - result.importances_std[i] < 0]
Handling Class Imbalance
Rebalance bootstrap samples per tree instead of naive oversampling of the full dataset.
python
from sklearn.ensemble import RandomForestClassifier# class_weight='balanced_subsample' recomputes weights on EACH bootstrap draw,# which better reflects the imbalance actually seen by each tree than# 'balanced' (computed once on the full training set).rf_imb = RandomForestClassifier( n_estimators=400, class_weight='balanced_subsample', min_samples_leaf=5, # larger leaves reduce variance from the rare class n_jobs=-1, random_state=42,)rf_imb.fit(X_train, y_train)# Threshold tuning on the minority class often beats resampling entirelyprobs = rf_imb.predict_proba(X_test)[:, 1]preds = (probs > 0.3).astype(int) # lower threshold favors recall
Incremental Growth with warm_start
Add trees to an existing forest without refitting from scratch, tracking OOB error as it stabilizes.
python
from sklearn.ensemble import RandomForestClassifierrf = RandomForestClassifier( n_estimators=50, warm_start=True, oob_score=True, n_jobs=-1, random_state=42)oob_curve = []for n in range(50, 501, 50): rf.n_estimators = n rf.fit(X_train, y_train) oob_curve.append((n, 1 - rf.oob_score_))# Plot oob_curve to find where error flattens -> stop adding trees there,# since extra trees beyond that point only cost compute, not accuracy.
Advanced Theory & Diagnostics
Concepts beyond the standard bagging/OOB intro.
- Bias-variance decomposition- Averaging B trees reduces variance by roughly 1/B only if trees are uncorrelated; feature randomness exists specifically to drive that correlation down
- Minimal cost-complexity pruning (ccp_alpha)- Applies post-hoc pruning per tree in the forest; rarely needed since averaging already controls variance, but shrinks model size for deployment
- Proximity matrix- Fraction of trees in which two samples land in the same leaf; usable as a learned similarity metric for clustering or outlier detection
- Quantile regression forests- Instead of averaging leaf targets, retain the full distribution of leaf samples to estimate prediction intervals (see skgarden/quantile-forest)
- max_samples- Caps the bootstrap sample size below 100% of the data; smaller draws increase tree diversity and can speed up training on huge datasets
- Impurity-based vs permutation importance bias- Impurity importance is computed on training data and is biased toward high-cardinality/continuous features even under pure noise
- Correlated features and importance splitting- When two features are highly correlated, RF splits importance credit between them, understating each one's individual signal
Pro Tip
Prefer permutation_importance from sklearn.inspection over the default feature_importances_ when features vary in cardinality or scale — impurity-based importance is biased toward high-cardinality and continuous features, even when they're not truly predictive.
Was this cheat sheet helpful?
Explore Topics
#RandomForest#RandomForestCheatSheet#DataScience#Intermediate#ClassifierWithScikitLearn#RegressorFeatureImportance#HyperparameterSearch#KeyConcepts#MachineLearning#CheatSheet#SkillVeris
