Polynomial and Multiple Regression
Simple linear regression models a target as a straight-line function of a single input. Real problems rarely cooperate: outcomes usually depend on many variables at once, and the relationship between an input and the target is often curved rather than linear. Multiple regression addresses the first limitation by adding more predictor variables to the same linear equation. Polynomial regression addresses the second by adding transformed versions of existing features (squares, cubes, interaction terms) as new columns, while still fitting a model that is linear in its coefficients. Both techniques use ordinary least squares under the hood — the trick is entirely in how the feature matrix is constructed before fitting.
Cricket analogy: Predicting a batsman's score using only balls faced is simple regression; adding pitch condition, opponent bowling attack, and weather (multiple regression) or curving the trend for late-innings acceleration (polynomial) still uses the same least-squares fit, just richer inputs.
Multiple Linear Regression
In multiple regression the prediction is y = b0 + b1*x1 + b2*x2 + ... + bn*xn. Each coefficient bi represents the expected change in y for a one-unit increase in xi, holding all other predictors constant — this 'holding constant' interpretation is what makes multiple regression more informative than running separate simple regressions for each feature. It also lets the model separate the effect of correlated inputs, though severe multicollinearity (predictors that are almost linear combinations of each other) makes individual coefficients unstable even when overall predictions remain accurate.
Cricket analogy: A coefficient for 'overs remaining' in a run-rate model shows how runs change per over when strike rate and wickets in hand are held fixed, letting analysts isolate that variable's real effect, unless overs and wickets are so tied together (multicollinearity) that neither estimate is trustworthy.
Polynomial Regression via Feature Expansion
Polynomial regression does not change the underlying algorithm at all; it changes the input. Given a single feature x, you generate new columns x^2, x^3, and so on, then hand the expanded matrix to ordinary linear regression. The model y = b0 + b1*x + b2*x^2 is linear in the parameters b0, b1, b2 even though it is a curve in x — this is why it is still called 'linear' regression in the statistical sense. Higher-degree polynomials can fit increasingly wiggly curves, but degree is a hyperparameter that trades bias for variance: too low underfits a genuine curve, too high starts chasing noise between data points, especially near the edges of the training range (Runge's phenomenon).
Cricket analogy: Modeling a bowler's economy rate as it curves upward after over 40 (death overs) requires adding over-number-squared as a new column, still fit with ordinary least squares; picking too low a degree misses the death-overs spike, too high one chases noise like a single expensive over.
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
# Single feature with a curved relationship
rng = np.random.default_rng(0)
X = rng.uniform(-3, 3, size=(200, 1))
y = 0.5 * X[:, 0] ** 2 - 2 * X[:, 0] + 3 + rng.normal(0, 1, 200)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
for degree in [1, 2, 5]:
model = make_pipeline(PolynomialFeatures(degree=degree, include_bias=False), LinearRegression())
model.fit(X_train, y_train)
preds = model.predict(X_test)
rmse = mean_squared_error(y_test, preds) ** 0.5
print(f"degree={degree} test RMSE={rmse:.3f}")
# Multiple regression with several raw features
X_multi = rng.normal(size=(200, 3))
y_multi = 2.0 * X_multi[:, 0] - 1.5 * X_multi[:, 1] + 0.3 * X_multi[:, 2] + rng.normal(0, 0.5, 200)
multi_model = LinearRegression().fit(X_multi, y_multi)
print("coefficients:", multi_model.coef_)Think of polynomial regression as linear regression wearing a disguise: you are not teaching the algorithm to draw curves, you are handing it curved features and letting it draw a straight line through a reshaped space. PCA, kernel methods, and basis-function models all share this same trick of transforming the input rather than the algorithm.
PolynomialFeatures on multiple inputs also creates interaction terms (x1*x2, x1*x2^2, ...), and the number of generated columns grows combinatorially with both the degree and the number of original features. A degree-4 expansion of 10 features can produce hundreds of columns, inviting overfitting and multicollinearity unless paired with regularization or feature selection.
Choosing Degree and Diagnosing Fit
The right polynomial degree is found the same way any hyperparameter is: by comparing validation error across candidate degrees, not by eyeballing the training fit. A useful diagnostic is to plot training and validation error as degree increases — training error falls monotonically while validation error typically forms a U-shape, and the minimum of that U marks a reasonable degree. In practice, analysts often prefer adding a small number of domain-informed polynomial or interaction terms over blindly expanding to high degree, since interpretability degrades quickly as the feature count grows.
Cricket analogy: Choosing how many overs-based polynomial terms to model economy rate is done by checking validation error across degrees on held-out matches, not by admiring a perfect fit to one series, and often a simple death-overs indicator beats a wild high-degree curve.
- Multiple regression adds more predictor variables; polynomial regression adds transformed (powered) versions of existing predictors — both remain linear in their coefficients.
- Coefficients in multiple regression represent an effect 'holding other predictors constant', which breaks down under strong multicollinearity.
- PolynomialFeatures generates powers and interaction terms; column count grows combinatorially with degree and original feature count.
- Degree is a bias-variance hyperparameter: too low underfits curvature, too high overfits and extrapolates wildly outside the training range.
- Select degree via validation error, not training error, since training error decreases monotonically with degree.
- High-degree expansions usually need regularization (ridge/lasso) to stay well-behaved.
Practice what you learned
1. Why is polynomial regression still considered a form of linear regression?
2. In multiple linear regression, how should a coefficient bi be interpreted?
3. What typically happens to training error as polynomial degree increases?
4. What is a major practical risk of expanding many original features to high polynomial degree?
5. Why do high-degree polynomial fits often behave poorly near the edges of the training data range?
Was this page helpful?
You May Also Like
Linear Regression Explained
Understand how linear regression fits a straight-line relationship between features and a continuous target, and how it is trained and evaluated.
Regularization: Ridge and Lasso
Explains how L2 (ridge) and L1 (lasso) penalties shrink regression coefficients to combat overfitting, and why lasso can perform feature selection.
Evaluating Regression Models
Surveys the core metrics for assessing regression quality — MAE, MSE, RMSE, and R-squared — and explains when each is the right lens for model performance.
Overfitting and Underfitting
The two failure modes of model fitting — memorizing noise in training data versus failing to capture real patterns — and the diagnostic and mitigation techniques for each.