SVM — Kernel Trick and Maximum Margin — Machine Learning with Scikit-learn

Support Vector Machines (SVMs) are classification algorithms that find the hyperplane maximising the margin between classes — the width of the empty band that separates the positive from the negative examples. This maximum-margin principle gives SVMs strong theoretical guarantees: the margin directly controls the model's generalisation bound, and a wider margin means better generalisation regardless of the number of features. SVMs were the dominant classification algorithm before deep learning and remain highly competitive on small-to-medium datasets.

Their defining innovation is the kernel trick: by implicitly mapping inputs to a very high-dimensional (possibly infinite-dimensional) feature space using only inner products, SVMs can find non-linear decision boundaries in the original space without ever computing the high-dimensional features explicitly. This makes SVMs practically powerful for non-linear problems while remaining theoretically grounded in convex optimisation. Understanding kernels and the margin explains why SVMs work where logistic regression fails and why they require careful scaling and hyperparameter tuning.

Analogy🏏Cricket

🏏 Think of it like cricket: A batting coach who teaches only one shot — the forward defensive — gives advice with high bias: it is consistently wrong for deliveries that demand a drive or a pull, regardless of how much practice the batsman does. A coach who memorises every ball of the batsman's training career gives advice with high variance: he predicts each training ball perfectly but fails completely on new balls from a different bowler, because he learned the noise of that specific bowler rather than the underlying principles. The great coach finds the balance — teaching the core principles that generalise, without over-specifying for the particular training environment. Just as great coaching lies between the extremes, great ML models balance bias and variance.