How Do Decision Trees Work?
Understand how decision trees split data using Gini impurity and entropy, why they overfit, and how to train one in scikit-learn with an interview-ready guide.
Expected Interview Answer
A decision tree is a supervised model that makes predictions by recursively splitting the data into branches based on feature thresholds, forming a tree of if-then questions that lead to a prediction at each leaf.
At every node the algorithm chooses the feature and split point that best separates the data, using an impurity measure like Gini impurity or entropy (information gain) for classification, or variance reduction for regression. Splitting continues until a stopping rule is met — maximum depth, minimum samples per leaf, or pure nodes. Trees are easy to interpret and handle mixed feature types, but a single deep tree overfits easily, which is why pruning and ensembles like random forests are used.
- Highly interpretable and easy to visualize
- Handles both numerical and categorical features
- Requires little data preprocessing or scaling
- Captures non-linear relationships and interactions
- Forms the building block of powerful ensembles
AI Mentor Explanation
A captain sets fielding tactics with a chain of questions: is the batter right-handed? Is it a spinner bowling? Is the pitch turning? Each answer routes to a specific plan. A decision tree classifies the same way, splitting on one feature at a time until every path ends in a concrete decision about the delivery.
Step-by-Step Explanation
Step 1
Start at the root
Begin with the full dataset at the top node of the tree.
Step 2
Evaluate candidate splits
For each feature and threshold, measure how much it reduces impurity (Gini, entropy, or variance).
Step 3
Choose the best split
Pick the feature and cutoff that gives the greatest information gain or lowest impurity.
Step 4
Recurse on child nodes
Repeat the splitting process on each resulting branch of data.
Step 5
Stop and assign leaves
Halt on depth, min-samples, or purity rules, then assign each leaf a class or value.
What Interviewer Expects
- Understands impurity measures like Gini and entropy
- Can explain information gain driving splits
- Knows how overfitting happens and how pruning helps
- Distinguishes classification from regression trees
- Mentions ensembles as a remedy for high variance
Common Mistakes
- Confusing Gini impurity with entropy or using them interchangeably without understanding
- Not recognizing that deep trees overfit
- Forgetting that trees need no feature scaling
- Assuming a single tree is as accurate as an ensemble
- Ignoring stopping criteria and letting the tree grow unbounded
Best Answer (HR Friendly)
“A decision tree is like a flowchart of yes/no questions that leads to an answer. The model learns which questions to ask, in the best order, to sort data into groups — for example, deciding if a customer will churn based on their usage and plan.”
Code Example
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
model = DecisionTreeClassifier(
criterion='gini', max_depth=3, random_state=42
)
model.fit(X_train, y_train)
preds = model.predict(X_test)
print('Accuracy:', accuracy_score(y_test, preds))
print(export_text(model, feature_names=load_iris().feature_names))Follow-up Questions
- What is the difference between Gini impurity and entropy?
- How does pruning reduce overfitting in decision trees?
- How does a regression tree differ from a classification tree?
- Why do decision trees have high variance?
- How do random forests improve on a single tree?
MCQ Practice
1. Which criterion is commonly used to choose splits in a classification tree?
Gini impurity (or entropy/information gain) measures how mixed the classes are and guides the best split.
2. A very deep decision tree most likely suffers from:
Deep trees memorize the training data, capturing noise and generalizing poorly — a classic overfitting symptom.
3. Which preprocessing step is generally NOT required for decision trees?
Trees split on thresholds per feature, so scaling or normalization does not change the splits or results.
Flash Cards
What measure guides splits in a classification tree? — Impurity: Gini impurity or entropy (information gain).
Why do single deep trees overfit? — They keep splitting until they memorize noise, giving high variance on new data.
Do decision trees need feature scaling? — No — splits are threshold-based per feature, so scaling has no effect.
What is pruning? — Cutting back branches that add little predictive value to reduce overfitting.