Decision Trees Cheat Sheet
A reference for decision trees covering scikit-learn classifiers and regressors, splitting criteria like Gini and entropy, pruning, and feature importance.
1 PageBeginnerMar 5, 2026
Classifier with scikit-learn
Fit and visualize a decision tree.
python
from sklearn.tree import DecisionTreeClassifier, plot_treeimport matplotlib.pyplot as pltclf = DecisionTreeClassifier( criterion='gini', max_depth=5, min_samples_leaf=10, random_state=42)clf.fit(X_train, y_train)plt.figure(figsize=(12, 8))plot_tree(clf, feature_names=feature_names, class_names=class_names, filled=True)plt.show()
Splitting Criteria
The math used to choose each split.
python
# Gini impurity: G = 1 - sum(p_i^2) over classes# Entropy: H = -sum(p_i * log2(p_i))# Information Gain = H(parent) - weighted_avg(H(children))from sklearn.tree import DecisionTreeRegressorreg = DecisionTreeRegressor(criterion='squared_error', max_depth=4)reg.fit(X_train, y_train) # regression trees split to minimize variance (MSE)
Feature Importance
Inspect which features drove the splits.
python
importances = clf.feature_importances_for name, imp in sorted(zip(feature_names, importances), key=lambda x: -x[1]): print(f'{name}: {imp:.3f}')
Key Concepts
Core theory behind decision trees.
- Gini impurity- Probability of misclassifying a randomly chosen sample; 0 means a perfectly pure node
- Entropy- Information-theoretic impurity measure; higher entropy means more disorder within a node
- Pruning- Reduces overfitting via max_depth limits (pre-pruning) or cost-complexity pruning with ccp_alpha (post-pruning)
- max_depth / min_samples_leaf- Core hyperparameters that trade off tree complexity against overfitting risk
- Overfitting- Unconstrained trees can memorize training data perfectly but generalize poorly
Cost-Complexity Pruning Path
Compute the full ccp_alpha pruning path and pick the value that maximizes validation accuracy.
python
path = clf.cost_complexity_pruning_path(X_train, y_train)alphas, impurities = path.ccp_alphas, path.impuritiesscores = []for alpha in alphas: pruned = DecisionTreeClassifier(random_state=42, ccp_alpha=alpha).fit(X_train, y_train) scores.append(pruned.score(X_test, y_test))best_alpha = alphas[int(np.argmax(scores))]final_tree = DecisionTreeClassifier(random_state=42, ccp_alpha=best_alpha).fit(X_train, y_train)
Export Human-Readable Rules
Turn a fitted tree into readable if/else rules or a Graphviz diagram for stakeholders.
python
from sklearn.tree import export_text, export_graphvizprint(export_text(clf, feature_names=list(feature_names)))export_graphviz( clf, out_file='tree.dot', feature_names=feature_names, class_names=class_names, filled=True, rounded=True)# dot -Tpng tree.dot -o tree.png
Permutation Importance
A less biased alternative to impurity-based feature_importances_.
python
from sklearn.inspection import permutation_importanceresult = permutation_importance( clf, X_test, y_test, n_repeats=30, random_state=42, n_jobs=-1)for name, mean, std in sorted( zip(feature_names, result.importances_mean, result.importances_std), key=lambda x: -x[1]): print(f'{name}: {mean:.3f} +/- {std:.3f}')
Inspect the Decision Path
Trace exactly which nodes and thresholds produced a given prediction.
python
node_indicator = clf.decision_path(X_test)leaf_id = clf.apply(X_test)sample_id = 0node_ids = node_indicator.indices[ node_indicator.indptr[sample_id]: node_indicator.indptr[sample_id + 1]]for node_id in node_ids: if leaf_id[sample_id] == node_id: continue feature = clf.tree_.feature[node_id] threshold = clf.tree_.threshold[node_id] value = X_test[sample_id, feature] direction = '<=' if value <= threshold else '>' print(f'node {node_id}: X[{feature}] = {value:.2f} {direction} {threshold:.2f}')
Advanced Concepts
Deeper mechanics and variants beyond a default DecisionTreeClassifier.
- Surrogate splits- Backup split rules used by some implementations (e.g. rpart, not sklearn) to route samples with missing values
- min_impurity_decrease- Prevents a split unless it reduces impurity by at least this amount, acting as another pre-pruning lever
- class_weight- Set to 'balanced' to reweight the impurity criterion inversely proportional to class frequency for imbalanced data
- Monotonic constraints- monotonic_cst forces predictions to be non-decreasing/non-increasing in a feature, useful for regulated risk models
- Native categorical support- HistGradientBoostingClassifier/Regressor accept categorical_features directly, avoiding one-hot explosion that plain DecisionTreeClassifier requires
- Extremely Randomized Trees- ExtraTreesClassifier picks split thresholds randomly instead of optimally, trading per-tree accuracy for lower variance in ensembles
Pro Tip
A single unconstrained decision tree almost always overfits — tune max_depth, min_samples_leaf, or ccp_alpha cost-complexity pruning, or better yet use the tree only as a base learner inside a Random Forest or Gradient Boosting ensemble.
Was this cheat sheet helpful?
Explore Topics
#DecisionTrees#DecisionTreesCheatSheet#DataScience#Beginner#ClassifierWithScikitLearn#SplittingCriteria#FeatureImportance#KeyConcepts#DataStructures#MachineLearning#CheatSheet#SkillVeris
