ROC & AUC Cheat Sheet
How the ROC curve and AUC score measure binary classifier performance across thresholds, with plotting and interpretation using scikit-learn.
1 PageIntermediateMar 12, 2026
ROC Curve & AUC Score
Compute and plot the ROC curve.
python
from sklearn.metrics import roc_curve, roc_auc_score, RocCurveDisplayimport matplotlib.pyplot as plt# y_scores = predicted probabilities for the positive classy_scores = model.predict_proba(X_test)[:, 1]fpr, tpr, thresholds = roc_curve(y_test, y_scores)auc = roc_auc_score(y_test, y_scores)print(f"AUC: {auc:.3f}")RocCurveDisplay(fpr=fpr, tpr=tpr, roc_auc=auc).plot()plt.plot([0, 1], [0, 1], linestyle="--", label="Random classifier")plt.legend()plt.show()
Precision-Recall Curve
Better alternative for rare-positive-class problems.
python
from sklearn.metrics import precision_recall_curve, average_precision_score# Use PR curve instead of ROC when the positive class is rareprecision, recall, thresholds = precision_recall_curve(y_test, y_scores)ap = average_precision_score(y_test, y_scores)print(f"Average Precision: {ap:.3f}")
ROC Concepts
Key terms behind the ROC curve.
- ROC curve- plots True Positive Rate (recall) vs False Positive Rate at every threshold
- True Positive Rate (TPR)- TP / (TP + FN); same as recall/sensitivity
- False Positive Rate (FPR)- FP / (FP + TN); fraction of negatives incorrectly flagged
- AUC- area under the ROC curve; probability a random positive scores higher than a random negative
- AUC = 0.5- no better than random guessing
- AUC = 1.0- perfect separation between classes
- Threshold- the cutoff probability used to convert scores into class predictions
Interpretation Tips
How to read and apply ROC/AUC in practice.
- Threshold selection- pick the threshold on the curve closest to the top-left corner, or by business cost
- Comparing models- a higher AUC means better ranking of positives above negatives on average
- Imbalanced data caveat- ROC-AUC can look optimistic when negatives vastly outnumber positives
- PR-AUC alternative- preferred over ROC-AUC for rare-event/imbalanced classification tasks
- Not threshold-specific- AUC summarizes performance across all thresholds, not one operating point
Multi-Class ROC (One-vs-Rest)
Extend ROC/AUC to multi-class problems with per-class and averaged curves.
python
from sklearn.preprocessing import label_binarizefrom sklearn.metrics import roc_curve, roc_auc_scoreimport numpy as npclasses = [0, 1, 2]y_test_bin = label_binarize(y_test, classes=classes) # one-hot per classy_scores = model.predict_proba(X_test) # shape (n_samples, n_classes)# per-class ROC/AUCfor i, c in enumerate(classes): fpr, tpr, _ = roc_curve(y_test_bin[:, i], y_scores[:, i]) auc_i = roc_auc_score(y_test_bin[:, i], y_scores[:, i]) print(f"class {c}: AUC = {auc_i:.3f}")# macro and micro averaged multi-class AUC in one callmacro_auc = roc_auc_score(y_test, y_scores, multi_class="ovr", average="macro")weighted_auc = roc_auc_score(y_test, y_scores, multi_class="ovr", average="weighted")print(f"macro AUC: {macro_auc:.3f}, weighted AUC: {weighted_auc:.3f}")
Bootstrap Confidence Interval for AUC
Estimate uncertainty around a single AUC point estimate without assuming normality.
python
import numpy as npfrom sklearn.metrics import roc_auc_scorerng = np.random.RandomState(42)y_test_arr, y_scores_arr = np.array(y_test), np.array(y_scores)n_bootstraps = 2000boot_aucs = []for _ in range(n_bootstraps): idx = rng.randint(0, len(y_scores_arr), len(y_scores_arr)) if len(np.unique(y_test_arr[idx])) < 2: continue # skip resamples with only one class present boot_aucs.append(roc_auc_score(y_test_arr[idx], y_scores_arr[idx]))boot_aucs = np.array(boot_aucs)lower, upper = np.percentile(boot_aucs, [2.5, 97.5])print(f"AUC: {roc_auc_score(y_test_arr, y_scores_arr):.3f}, 95% CI: [{lower:.3f}, {upper:.3f}]")
Optimal Threshold via Youden's J
Find the point on the ROC curve that best balances sensitivity and specificity.
python
from sklearn.metrics import roc_curveimport numpy as npfpr, tpr, thresholds = roc_curve(y_test, y_scores)# Youden's J = TPR - FPR, maximized at the point closest to the top-left cornerj_scores = tpr - fprbest_idx = np.argmax(j_scores)best_threshold = thresholds[best_idx]print(f"Optimal threshold: {best_threshold:.3f}")print(f"TPR: {tpr[best_idx]:.3f}, FPR: {fpr[best_idx]:.3f}, J: {j_scores[best_idx]:.3f}")
Calibration Curve & Brier Score
Check whether predicted probabilities are trustworthy, not just well-ranked.
python
from sklearn.calibration import calibration_curve, CalibrationDisplayfrom sklearn.metrics import brier_score_lossimport matplotlib.pyplot as plt# a high AUC only means good ranking -- it says nothing about whether# predict_proba() outputs are close to true probabilitiesfrac_positives, mean_predicted = calibration_curve(y_test, y_scores, n_bins=10)brier = brier_score_loss(y_test, y_scores)print(f"Brier score (lower is better, 0 = perfect): {brier:.4f}")CalibrationDisplay(prob_true=frac_positives, prob_pred=mean_predicted, y_prob=y_scores).plot()plt.plot([0, 1], [0, 1], linestyle="--", color="gray")plt.show()
Advanced ROC/AUC Concepts
Terminology beyond the basic curve for rigorous model comparison.
- Partial AUC- AUC restricted to a clinically/operationally relevant FPR range (e.g. FPR < 0.1) when only that region matters
- DeLong's test- statistical test for whether two correlated ROC curves (same test set) have significantly different AUC
- Concordance index (c-index)- generalization of AUC to survival/time-to-event models with censored data
- Calibration vs. discrimination- AUC measures discrimination (ranking); calibration measures whether probability values themselves are accurate
- Threshold moving- deploying the model with a non-default decision threshold chosen from the ROC/PR curve instead of retraining
- Class imbalance sensitivity- unlike PR-AUC, ROC-AUC is insensitive to the positive/negative ratio, which is why it can look stable even as prevalence shifts
Pro Tip
For heavily imbalanced datasets (e.g. fraud detection with 1% positives), prefer the Precision-Recall AUC over ROC-AUC — ROC-AUC can stay deceptively high because the large number of true negatives dominates the false positive rate.
Was this cheat sheet helpful?
Explore Topics
#ROCAUC#ROCAUCCheatSheet#DataScience#Intermediate#ROC#Curve#AUC#Score#MachineLearning#CheatSheet#SkillVeris
