What is a Random Forest?
Learn what a random forest is, how bagging and feature randomness reduce variance, and how out-of-bag error and feature importance work in practice.
Expected Interview Answer
A random forest is an ensemble learning method that trains many decision trees on different bootstrapped samples of the data and random subsets of features, then combines their predictions by majority vote (classification) or averaging (regression) to produce a more accurate and stable result than any single tree.
Each tree in the forest is trained on a bootstrap sample (random sampling with replacement) of the training data, and at each split the algorithm only considers a random subset of features rather than all of them, a technique called feature bagging. This dual randomness decorrelates the individual trees, so their errors are less likely to be correlated, meaning the ensemble's averaged or voted prediction has substantially lower variance than any single deep, overfit tree. Individual trees can be grown deep and overfit their own bootstrap sample without hurting the ensemble, because averaging over many decorrelated trees cancels out their individual noise. Random forests also provide a built-in feature importance measure (based on impurity reduction or permutation importance) and an out-of-bag error estimate using the roughly one-third of samples not included in each tree's bootstrap, avoiding the need for a separate validation set.
- Substantially reduces variance and overfitting compared to a single decision tree
- Handles nonlinear relationships and feature interactions without manual engineering
- Provides built-in feature importance rankings
- Robust to outliers and requires minimal hyperparameter tuning to get solid results
- Out-of-bag error gives a free validation-like estimate without a separate holdout set
AI Mentor Explanation
A random forest is like a selection panel where each selector watches a different, overlapping subset of a player's matches and considers only certain aspects of their game before voting. Averaging many such independent, imperfect opinions produces a far more reliable selection than trusting one selector's judgment, which could be skewed by the matches they watched.
Step-by-Step Explanation
Step 1
Create bootstrap samples
For each tree, draw a random sample of the training data with replacement (a bootstrap sample), roughly the same size as the original set.
Step 2
Grow a decision tree on each sample
Train a full, typically unpruned decision tree on each bootstrap sample.
Step 3
Randomize feature selection at each split
At every node, only consider a random subset of features (not all of them) when choosing the best split, decorrelating the trees.
Step 4
Repeat for many trees
Build hundreds of such decorrelated trees, each slightly different due to bootstrapping and feature randomness.
Step 5
Aggregate predictions
Combine all trees' outputs via majority vote for classification or averaging for regression to form the final prediction.
Step 6
Use out-of-bag samples for validation
Evaluate each tree on the roughly one-third of data it never saw (out-of-bag), giving a built-in generalization estimate.
What Interviewer Expects
- Explains bootstrap sampling (bagging) and feature randomness together
- Understands why averaging decorrelated trees reduces variance
- Mentions the out-of-bag error estimate
- Knows random forest provides feature importance
- Can compare random forest to a single decision tree and to gradient boosting
Common Mistakes
- Confusing random forest (bagging) with gradient boosting (sequential correction)
- Forgetting the random feature subset step, only mentioning bootstrap sampling
- Claiming random forests never overfit, when they can with too few trees or too little diversity
- Not knowing what out-of-bag error is
- Assuming individual trees in the forest are pruned when they are typically grown deep
Best Answer (HR Friendly)
“A random forest builds many different decision trees, each trained on a slightly different random slice of the data, and then combines all their predictions by voting or averaging. This teamwork approach is far more accurate and stable than relying on just one tree, because the trees' individual mistakes tend to cancel each other out.”
Code Example
from sklearn.ensemble import RandomForestClassifier
import numpy as np
forest = RandomForestClassifier(
n_estimators=300,
max_features="sqrt", # random feature subset per split
oob_score=True, # out-of-bag validation estimate
random_state=42
)
forest.fit(X_train, y_train)
print("OOB score:", forest.oob_score_)
print("Test accuracy:", forest.score(X_test, y_test))
importances = forest.feature_importances_
top5 = np.argsort(importances)[::-1][:5]
print("Top 5 features:", top5)Follow-up Questions
- How does bootstrap sampling combined with feature randomness decorrelate the trees?
- What is out-of-bag error and why is it useful?
- How does random forest differ from gradient boosting?
- Why can individual trees in a random forest be grown deep without hurting the ensemble?
- How do you interpret random forest feature importance scores?
MCQ Practice
1. What two sources of randomness does a random forest use to decorrelate its trees?
Random forests combine bootstrap sampling (bagging) of training examples with random feature subsets at each split to decorrelate individual trees.
2. How does a random forest combine predictions from its individual trees for classification?
For classification, random forest predictions are combined via majority vote across all trees in the ensemble.
3. What is the out-of-bag error estimate?
Since each tree is trained on a bootstrap sample, roughly one-third of data is left out; evaluating on that out-of-bag portion gives a free validation estimate.
Flash Cards
What is a random forest? — An ensemble of decision trees trained on bootstrapped samples with random feature subsets, combined by voting or averaging.
Why does averaging many decorrelated trees reduce variance? — Individual trees' overfitting errors are largely uncorrelated, so averaging cancels much of that noise out.
What is out-of-bag error? — A generalization estimate computed from the roughly one-third of samples excluded from each tree's bootstrap sample.
How does random forest differ from gradient boosting? — Random forest trains trees independently in parallel (bagging); gradient boosting trains trees sequentially, each correcting the prior tree's errors.