Random Forest
Random Forest is an ensemble learning algorithm that builds many decision trees on randomly resampled data and random feature subsets, then aggregates their predictions by voting or averaging to produce a more accurate and robust model.
Definition
Random Forest is an ensemble learning algorithm that builds many decision trees on randomly resampled data and random feature subsets, then aggregates their predictions by voting or averaging to produce a more accurate and robust model.
Overview
Random Forest, introduced by Leo Breiman in 2001, combines two sources of randomness to build a diverse collection of decision trees whose errors are largely uncorrelated. First, each tree is trained on a bootstrap sample — a random sample of the training data drawn with replacement — a technique known as bagging. Second, at each split within each tree, only a random subset of features is considered as candidates, rather than all available features, which further decorrelates the trees by preventing a few dominant features from producing nearly identical splits across the entire forest. For classification tasks, the forest's prediction is typically the majority vote across all trees; for regression, it is the average of all trees' predictions. Because individual decision trees tend to overfit their training data, aggregating many decorrelated trees substantially reduces variance without a large increase in bias, generally producing far more robust and accurate predictions than any single tree while remaining relatively resistant to overfitting even with many trees. Random Forests are valued for being easy to use with minimal hyperparameter tuning, handling both numerical and categorical features, tolerating missing values and outliers reasonably well, and requiring no feature scaling. They also naturally provide feature importance scores, useful for interpretability and feature selection, and out-of-bag error estimates, which give an unbiased estimate of test error using the training data alone, without a separate validation set. While gradient boosting methods like XGBoost frequently edge out Random Forests on raw predictive accuracy in competitions, Random Forest remains a widely used, dependable baseline across finance, healthcare, and general tabular-data applications due to its simplicity and resistance to overfitting.
Key Concepts
- Bagging — trains each tree on a bootstrap-resampled subset of the training data
- Random feature subsetting at each split decorrelates trees further
- Majority voting (classification) or averaging (regression) across all trees
- Reduces overfitting compared to a single deep decision tree
- Built-in feature importance scores for interpretability
- Out-of-bag error estimation without a separate validation set
- Minimal preprocessing required — handles mixed feature types, no scaling needed
- Relatively robust to outliers and missing data