Linear Regression Cheat Sheet
A reference for linear regression covering scikit-learn implementation, the normal equation, regularized variants, and key statistical assumptions to check.
1 PageBeginnerMar 10, 2026
Fitting with scikit-learn
Train and evaluate an ordinary least squares model.
python
from sklearn.linear_model import LinearRegressionfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import mean_squared_error, r2_scoreX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)model = LinearRegression()model.fit(X_train, y_train)y_pred = model.predict(X_test)print('R2:', r2_score(y_test, y_pred))print('MSE:', mean_squared_error(y_test, y_pred))print('Coefficients:', model.coef_, 'Intercept:', model.intercept_)
Normal Equation
Closed-form solution without gradient descent.
python
# Closed-form solution: beta = (X^T X)^-1 X^T yimport numpy as npX_b = np.c_[np.ones((len(X), 1)), X] # add bias/intercept columnbeta = np.linalg.inv(X_b.T @ X_b) @ X_b.T @ y
Regularized Variants
Ridge, Lasso, and Elastic Net regression.
python
from sklearn.linear_model import Ridge, Lasso, ElasticNetridge = Ridge(alpha=1.0).fit(X_train, y_train) # L2 penaltylasso = Lasso(alpha=0.1).fit(X_train, y_train) # L1 penalty, can zero out coefficientselastic = ElasticNet(alpha=0.1, l1_ratio=0.5).fit(X_train, y_train) # mix of L1 and L2
Key Concepts
Core theory behind linear regression.
- Ordinary Least Squares- Minimizes the sum of squared residuals between predicted and actual values
- R-squared- Proportion of variance in the target explained by the model; 1.0 is a perfect fit
- Multicollinearity- High correlation between features inflates coefficient variance; check with VIF
- Regularization- Ridge (L2) shrinks coefficients smoothly, Lasso (L1) can zero them out entirely
- Homoscedasticity- Assumption that residual variance stays constant across all predicted values
- Residual plot- Plot of residuals vs. predictions used to visually check assumption violations
Gradient Descent From Scratch
Implement batch gradient descent to fit weights without a closed-form solve.
python
import numpy as npdef fit_gd(X, y, lr=0.05, n_iters=2000): n, d = X.shape X_b = np.c_[np.ones(n), X] theta = np.zeros(d + 1) for i in range(n_iters): preds = X_b @ theta error = preds - y grad = (2 / n) * (X_b.T @ error) theta -= lr * grad if i % 500 == 0: mse = np.mean(error ** 2) print(f'iter {i}: mse={mse:.4f}') return theta # theta[0] is intercept, theta[1:] are coefficients
Statistical Inference with statsmodels
Get p-values, confidence intervals, and F-statistics that scikit-learn doesn't expose.
python
import statsmodels.api as smX_sm = sm.add_constant(X_train) # adds intercept columnols = sm.OLS(y_train, X_sm).fit()print(ols.summary()) # coef, std err, t, P>|t|, [0.025 0.975]print('95% CI:\n', ols.conf_int(alpha=0.05))print('F-statistic p-value:', ols.f_pvalue)
Variance Inflation Factor
Quantify multicollinearity per feature before trusting individual coefficients.
python
import pandas as pdfrom statsmodels.stats.outliers_influence import variance_inflation_factorX_const = sm.add_constant(X_train)vif = pd.DataFrame({ 'feature': X_const.columns, 'VIF': [variance_inflation_factor(X_const.values, i) for i in range(X_const.shape[1])]})print(vif.sort_values('VIF', ascending=False))# VIF > 5-10 signals problematic multicollinearity for that feature
Robust Regression for Outliers
Downweight or ignore outliers that would otherwise dominate an OLS fit.
python
from sklearn.linear_model import HuberRegressor, RANSACRegressorhuber = HuberRegressor(epsilon=1.35).fit(X_train, y_train) # loss ~quadratic near 0, linear past epsilon*sigmaransac = RANSACRegressor(random_state=42).fit(X_train, y_train)inlier_mask = ransac.inlier_mask_print('Inlier ratio:', inlier_mask.mean())
Advanced Diagnostic Tests
Formal checks for OLS assumption violations beyond eyeballing a residual plot.
- Durbin-Watson- Statistic near 2 indicates no autocorrelation in residuals; near 0 or 4 signals positive/negative autocorrelation
- Breusch-Pagan test- Formal hypothesis test for heteroscedasticity; low p-value rejects constant-variance assumption
- Cook's distance- Measures how much removing a single observation would change the fitted coefficients; flags influential outliers
- Q-Q plot- Plots residual quantiles against a normal distribution to check the normality-of-errors assumption
- Leverage (hat values)- Diagonal of the hat matrix; high-leverage points have extreme predictor values that can distort the fit
- Condition number- Large values (>30) from sm.OLS summary indicate numerically unstable, ill-conditioned design matrices
Pro Tip
Don't rely on R-squared alone to judge model fit — always plot residuals against predicted values, since a high R-squared can still hide non-linearity or heteroscedasticity that biases your standard errors and confidence intervals.
Was this cheat sheet helpful?
Explore Topics
#LinearRegression#LinearRegressionCheatSheet#DataScience#Beginner#FittingWithScikitLearn#NormalEquation#RegularizedVariants#KeyConcepts#MachineLearning#CheatSheet#SkillVeris
