What is a Decision Tree?
Learn what a decision tree is, how splitting criteria like Gini impurity work, why unpruned trees overfit, and how they power random forests.
Expected Interview Answer
A decision tree is a supervised learning model that predicts an outcome by repeatedly splitting the data on feature thresholds, forming a tree of if/else decision rules that route each example down to a leaf holding the final prediction.
Starting at the root node, the algorithm evaluates every feature and possible split point, choosing the one that best separates the data according to a criterion like Gini impurity or information gain (entropy) for classification, or variance reduction for regression. This splitting process repeats recursively on each resulting subset, building deeper branches until a stopping condition is met, such as reaching a maximum depth, a minimum number of samples per leaf, or a pure node. Decision trees are highly interpretable because you can trace the exact sequence of if/else rules that led to any prediction, but a single unconstrained tree tends to overfit badly, since it can keep splitting until every leaf contains just one training example. Pruning, setting a maximum depth, or requiring a minimum number of samples per split constrains complexity, and ensembling many trees (random forest, gradient boosting) dramatically improves accuracy and stability over a single tree.
- Highly interpretable — you can trace the exact decision path for any prediction
- Handles both numerical and categorical features without heavy preprocessing
- Naturally captures nonlinear relationships and feature interactions
- Requires no feature scaling, unlike distance-based algorithms
- Serves as the foundational building block for powerful ensembles like random forest and XGBoost
AI Mentor Explanation
A decision tree is like a captain's field-setting flowchart: first check if the batsman is left- or right-handed, then check the pitch condition, then check the current over, routing to a specific fielding plan at each branch. Following the flowchart from the trunk to a leaf gives one clear, traceable recommendation, exactly like tracing a prediction through a trained tree's splits.
Step-by-Step Explanation
Step 1
Start at the root with all data
The full training set begins at the root node before any splitting has occurred.
Step 2
Find the best split
Evaluate every feature and threshold, selecting the split that best reduces impurity (Gini/entropy for classification, variance for regression).
Step 3
Partition the data
Split the current node's examples into child nodes based on the chosen feature threshold.
Step 4
Recurse on each child
Repeat the best-split search independently on each resulting subset, building deeper branches.
Step 5
Apply stopping criteria
Stop splitting a branch when it reaches max depth, minimum samples per leaf, or a pure (single-class) node.
Step 6
Prune or constrain to avoid overfitting
Limit tree depth, require minimum samples per split, or prune post-hoc to prevent the tree from memorizing training noise.
What Interviewer Expects
- Explains the recursive splitting process and stopping criteria
- Names splitting criteria like Gini impurity, entropy/information gain, or variance reduction
- Understands that unconstrained trees overfit and how to control depth
- Connects decision trees to ensemble methods (random forest, gradient boosting)
- Recognizes decision trees' interpretability advantage
Common Mistakes
- Not mentioning any regularization technique like max depth or min samples per leaf
- Confusing Gini impurity with entropy without knowing both are valid split criteria
- Claiming decision trees require feature scaling
- Assuming a single decision tree is always as accurate as an ensemble
- Forgetting decision trees can be used for both classification and regression
Best Answer (HR Friendly)
“A decision tree is a model that makes predictions by asking a series of yes/no questions about the data, like a flowchart, until it reaches a final answer. It's easy to understand because you can trace exactly which questions led to each decision, though a single tree can be prone to overfitting if left unchecked.”
Code Example
from sklearn.tree import DecisionTreeClassifier, export_text
tree = DecisionTreeClassifier(
criterion="gini",
max_depth=4,
min_samples_leaf=10,
random_state=42
)
tree.fit(X_train, y_train)
print("Train accuracy:", tree.score(X_train, y_train))
print("Test accuracy:", tree.score(X_test, y_test))
print(export_text(tree, feature_names=list(X_train.columns)))Follow-up Questions
- What is the difference between Gini impurity and entropy as split criteria?
- How do you prevent a decision tree from overfitting?
- How does a decision tree handle regression versus classification tasks?
- What is pruning and how does it improve generalization?
- How does a random forest improve on a single decision tree?
MCQ Practice
1. What determines the best split at each node in a decision tree?
Decision trees select the split that best reduces impurity, measured by criteria like Gini impurity, entropy/information gain, or variance reduction.
2. What is a major weakness of a single, unconstrained decision tree?
An unconstrained tree can keep splitting until each leaf contains very few or single examples, memorizing noise and overfitting badly.
3. Which technique helps reduce a decision tree's tendency to overfit?
Constraining tree growth with a minimum samples per leaf (or maximum depth) prevents the tree from fitting every training noise point.
Flash Cards
How does a decision tree make predictions? — By routing an example through a series of if/else splits on feature thresholds down to a leaf holding the prediction.
Name two splitting criteria used in decision trees. — Gini impurity and information gain (entropy) for classification; variance reduction for regression.
Why do unconstrained decision trees overfit? — They can keep splitting until every leaf is pure or contains very few examples, memorizing training noise.
What ensemble methods build on decision trees? — Random forest (bagging of trees) and gradient boosting (sequential trees correcting prior errors).