Linear Regression — OLS and Assumptions — Machine Learning with Scikit-learn

Linear regression is the foundational supervised regression algorithm, modelling the relationship between input features and a continuous target as a weighted sum of the features plus an intercept. It is the starting point for all of regression not because it is always the best predictor, but because it is interpretable, computationally cheap, and analytically understood — its assumptions are explicit, its failure modes are diagnosable, and more complex methods are often understood relative to it.

Ordinary Least Squares (OLS) is the standard fitting method: it finds the coefficient vector that minimises the sum of squared residuals, the differences between the predicted and actual target values. This minimisation has an exact closed-form solution, making OLS the only major ML algorithm that does not require iterative optimisation. Understanding OLS deeply — what it minimises, when its assumptions hold, and what breaks when they do not — is the lens through which all regression is understood.

Analogy🏏Cricket

🏏 Think of it like cricket: A batting coach who teaches only one shot — the forward defensive — gives advice with high bias: it is consistently wrong for deliveries that demand a drive or a pull, regardless of how much practice the batsman does. A coach who memorises every ball of the batsman's training career gives advice with high variance: he predicts each training ball perfectly but fails completely on new balls from a different bowler, because he learned the noise of that specific bowler rather than the underlying principles. The great coach finds the balance — teaching the core principles that generalise, without over-specifying for the particular training environment. Just as great coaching lies between the extremes, great ML models balance bias and variance.