Evaluating Regression Models
A regression model that has finished training still needs to be judged, and unlike classification there is no single 'accuracy' number that obviously applies — predictions are continuous, so evaluation must measure how far off predictions are on average, in a way that is meaningful to the problem at hand. Several standard metrics exist, each making a different tradeoff between interpretability, sensitivity to large errors, and scale-independence, and choosing the right one (or combination) is itself part of good modeling practice.
Cricket analogy: Unlike judging a bowler by a simple wickets-taken count, evaluating a run-prediction model needs a metric measuring how far off each predicted score was on average, since 'close but not exact' still counts as informative in a way a binary accuracy number can't capture.
Mean Absolute Error and Mean Squared Error
Mean Absolute Error (MAE) averages the absolute difference between predicted and actual values: MAE = mean(|y_true - y_pred|). It is expressed in the same units as the target and treats all errors proportionally to their size, making it robust to outliers and easy to explain to non-technical stakeholders. Mean Squared Error (MSE) instead averages the squared differences: MSE = mean((y_true - y_pred)^2). Squaring penalizes large errors disproportionately more than small ones, which is desirable when large mistakes are especially costly, but it also means a handful of outliers can dominate the metric and make model comparisons misleading if outliers are not handled deliberately.
Cricket analogy: MAE averages how many runs off each score prediction was, in plain runs, robust even if one freak 150-run innings skews things; MSE squares those errors, so a single wildly missed prediction like an unexpected double century dominates the metric.
RMSE and R-squared
Root Mean Squared Error (RMSE) is simply the square root of MSE, which brings the metric back into the original units of the target while retaining MSE's sensitivity to large errors — this combination of interpretability and outlier sensitivity makes RMSE one of the most commonly reported regression metrics. R-squared (coefficient of determination) instead measures the proportion of variance in the target explained by the model: R^2 = 1 - (SS_residual / SS_total), where SS_total is the variance of y around its mean. An R^2 of 1.0 means the model explains all variance; an R^2 of 0 means the model is no better than always predicting the mean; and R^2 can go negative when a model performs worse than that trivial baseline, which happens routinely when evaluating on unseen test data with a poorly generalizing model.
Cricket analogy: RMSE brings the error back into runs while still punishing a wildly wrong prediction hard, making it the go-to reported metric; R-squared instead tells you what fraction of the variation in team scores your model explains versus just guessing the season average every time.
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
rng = np.random.default_rng(2)
X = rng.normal(size=(300, 4))
true_coef = np.array([1.5, -2.0, 0.0, 3.0])
y = X @ true_coef + 5 + rng.normal(0, 2.0, 300)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2)
model = LinearRegression().fit(X_train, y_train)
preds = model.predict(X_test)
mae = mean_absolute_error(y_test, preds)
mse = mean_squared_error(y_test, preds)
rmse = mse ** 0.5
r2 = r2_score(y_test, preds)
print(f"MAE: {mae:.3f}")
print(f"MSE: {mse:.3f}")
print(f"RMSE: {rmse:.3f}")
print(f"R^2: {r2:.3f}")
# A naive baseline that always predicts the training mean
baseline_preds = np.full_like(y_test, y_train.mean())
print("baseline R^2:", r2_score(y_test, baseline_preds))R-squared answers 'how much better is my model than always guessing the average?' — it is a relative metric, not an absolute error measure. Two models can have identical R^2 on datasets with very different scales and noise levels, so R^2 alone should rarely be the only number reported; pairing it with RMSE or MAE gives both relative and absolute pictures of performance.
A common mistake is comparing RMSE or MAE values across different datasets or different target transformations (e.g., raw price vs. log price) as if they were directly comparable — these metrics are scale-dependent, so an RMSE of 500 could be excellent or terrible depending entirely on the typical magnitude of the target variable.
Picking the Right Metric for the Problem
The choice of metric should reflect the cost structure of the real-world problem. If large errors are disproportionately costly — a delivery time estimate off by an hour is much worse than five estimates off by twelve minutes each — RMSE or MSE better reflects that asymmetry. If all errors matter roughly in proportion to their size, MAE is simpler and more robust to occasional extreme outliers. R-squared is useful for communicating overall explanatory power to stakeholders but should be reported alongside an absolute-error metric so that 'good' isn't just relative to a weak baseline.
Cricket analogy: If a wildly wrong score prediction (missing a collapse by 80 runs) is far costlier than several small misses, RMSE better reflects that asymmetry the way a captain weighs a disastrous batting-order call more heavily than several minor ones; if all errors matter proportionally, MAE is simpler and R-squared should accompany either to show it beats guessing the season average.
- MAE averages absolute errors, is in the target's units, and is robust to outliers.
- MSE averages squared errors, penalizing large errors more heavily but sensitive to outliers.
- RMSE is the square root of MSE, combining target-unit interpretability with sensitivity to large errors.
- R-squared measures the proportion of target variance explained relative to always predicting the mean, and can be negative.
- Metric choice should reflect the real-world cost of large vs. small errors.
- Report an absolute-error metric alongside R-squared, since R^2 alone does not convey error magnitude.
Practice what you learned
1. Which regression metric penalizes large errors disproportionately more than small errors due to squaring?
2. What does an R-squared value of 0 indicate about a regression model?
3. Why might RMSE be preferred over MSE when reporting model performance to stakeholders?
4. Under what condition can R-squared become negative?
5. Why is it problematic to directly compare RMSE values across two models trained on different target scales (e.g., house prices in dollars vs. house prices in thousands of dollars)?
Was this page helpful?
You May Also Like
Linear Regression Explained
Understand how linear regression fits a straight-line relationship between features and a continuous target, and how it is trained and evaluated.
Polynomial and Multiple Regression
Extends simple linear regression to multiple predictors and curved relationships, showing how feature expansion lets a linear model fit non-linear patterns.
Confusion Matrix and Classification Metrics
A structured breakdown of correct and incorrect predictions by class, forming the foundation for accuracy, precision, recall, and other classification metrics.
Overfitting and Underfitting
The two failure modes of model fitting — memorizing noise in training data versus failing to capture real patterns — and the diagnostic and mitigation techniques for each.