Linear Regression Explained
Linear regression is one of the oldest and most fundamental supervised learning algorithms, used to predict a continuous numeric target as a weighted sum of input features. For a single feature it fits a straight line; for multiple features it fits a hyperplane. The model takes the form y = w1*x1 + w2*x2 + ... + wn*xn + b, where each weight wi represents how much the predicted target changes for a one-unit increase in that feature (holding all other features constant), and b is the intercept — the predicted value when all features are zero. Despite its simplicity, linear regression remains widely used because it is fast to train, easy to interpret, and often a strong, hard-to-beat baseline on genuinely linear relationships.
Cricket analogy: Predicting a batter's final score as base runs (intercept) plus a weight times balls faced plus a weight times boundaries hit is a straight-line model, much like Virat Kohli's scoring rate estimated as a weighted sum of familiar inputs rather than a complex simulation.
How the Model Is Fit: Minimizing Squared Error
Linear regression is typically fit by finding the weights that minimize the mean squared error (MSE) between predicted and actual target values across the training data: MSE = (1/n) * sum((y_actual - y_predicted)^2). Squaring the errors before averaging penalizes large errors disproportionately more than small ones and ensures positive and negative errors don't cancel out. For smaller datasets, this optimal set of weights can be computed exactly using a closed-form matrix formula (the 'normal equation'); for larger datasets, or when the closed-form solution is computationally expensive, gradient descent is used instead, iteratively nudging the weights in the direction that reduces MSE fastest until convergence.
Cricket analogy: Minimizing squared error punishes one wildly wrong prediction, like missing a Buttler blitz by 80 runs, more than small misses; small datasets can be solved exactly, but a whole league's ball-by-ball data is too large, so weights are found iteratively, like gradient descent.
Assumptions and Interpreting Coefficients
Linear regression rests on several assumptions that, when violated, degrade its reliability: linearity (the true relationship between features and target is approximately linear), independence of errors (residuals aren't correlated with each other, which matters especially for time series), homoscedasticity (the spread of residuals is roughly constant across all predicted values, rather than fanning out), and low multicollinearity (features aren't highly correlated with each other, which otherwise makes individual coefficient estimates unstable and hard to interpret). When these hold reasonably well, the fitted coefficients are directly interpretable: a coefficient of 2.5 on 'square footage' means each additional square foot is associated with a 2.5-unit increase in predicted price, holding other features constant.
Cricket analogy: A model predicting runs only holds up if the relationship stays roughly linear, errors between batters aren't correlated, scoring variance stays steady, and features like strike rate and boundaries aren't too correlated; a coefficient of 2.5 on balls faced means 2.5 more runs per ball, other things equal.
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
rng = np.random.default_rng(42)
sqft = rng.uniform(500, 3500, 200)
bedrooms = rng.integers(1, 6, 200)
price = 50_000 + 120 * sqft + 8_000 * bedrooms + rng.normal(0, 15_000, 200)
X = np.column_stack([sqft, bedrooms])
X_train, X_test, y_train, y_test = train_test_split(X, price, test_size=0.2, random_state=42)
model = LinearRegression()
model.fit(X_train, y_train)
print('coefficients (sqft, bedrooms):', model.coef_.round(2))
print('intercept:', model.intercept_.round(2))
preds = model.predict(X_test)
print('RMSE:', mean_squared_error(y_test, preds, squared=False).round(2))
print('R^2:', r2_score(y_test, preds).round(3))R-squared (the coefficient of determination) tells you what fraction of the variance in the target is explained by the model, ranging roughly from 0 (no better than predicting the mean) to 1 (perfect fit). An R^2 of 0.75 means the model explains 75% of the variability in the target — but a high R^2 alone doesn't guarantee the model generalizes well or that its assumptions hold.
A common misconception is that a large coefficient means a feature is more 'important'. Coefficient magnitude depends heavily on the feature's scale — a coefficient of 50,000 on a feature measured in miles vs. 50 on the same distance measured in meters reflects unit choice, not importance. Compare standardized coefficients (fit on scaled features) if you want to compare relative feature importance fairly.
- Linear regression predicts a continuous target as a weighted sum of features plus an intercept.
- It is typically fit by minimizing mean squared error, via a closed-form solution or gradient descent.
- Key assumptions include linearity, independent errors, homoscedasticity, and low multicollinearity.
- Coefficients are interpretable as the change in predicted target per one-unit change in a feature, holding others constant.
- R-squared measures the proportion of target variance explained by the model but doesn't guarantee generalization.
- Raw coefficient magnitude is scale-dependent and should not be used alone to judge feature importance.
Practice what you learned
1. What does linear regression predict?
2. What loss function does ordinary linear regression typically minimize?
3. Which of the following is NOT a standard assumption of linear regression?
4. Why is a large coefficient value not automatically an indicator of feature importance?
5. What does an R-squared value of 0.75 indicate?
Was this page helpful?
You May Also Like
Polynomial and Multiple Regression
Extends simple linear regression to multiple predictors and curved relationships, showing how feature expansion lets a linear model fit non-linear patterns.
Regularization: Ridge and Lasso
Explains how L2 (ridge) and L1 (lasso) penalties shrink regression coefficients to combat overfitting, and why lasso can perform feature selection.
Evaluating Regression Models
Surveys the core metrics for assessing regression quality — MAE, MSE, RMSE, and R-squared — and explains when each is the right lens for model performance.
Gradient Descent Explained
Understand how gradient descent iteratively adjusts model parameters to minimize a loss function, and how learning rate and variants like SGD affect convergence.