Support Vector Machines
A Support Vector Machine (SVM) is a supervised learning algorithm that classifies data by finding the hyperplane that separates classes with the largest possible margin. Unlike logistic regression, which fits a probabilistic boundary, an SVM is explicitly geometric: it searches for the decision boundary that maximizes the distance to the nearest points of each class. Those nearest points are called support vectors, because they alone determine where the boundary sits — every other point could be deleted without changing the solution. This margin-maximization objective tends to produce boundaries that generalize well, especially when the number of features is large relative to the number of samples, such as in text classification or bioinformatics.
Cricket analogy: An SVM field placement finds the boundary line that keeps the maximum distance from the nearest fielders on both sides of the pitch; only the closest fielders (support vectors) determine that boundary, so moving a fielder standing deep at third man wouldn't change the placement at all.
The Margin and Support Vectors
For linearly separable data, the SVM solves an optimization problem: minimize the norm of the weight vector subject to every point being correctly classified with a margin of at least 1. Points that lie exactly on the margin boundary are the support vectors. A larger margin corresponds to a smaller weight vector norm, which is why SVM training is framed as minimizing ||w||^2 rather than directly minimizing misclassifications. In practice, real data is rarely perfectly separable, so a 'soft margin' formulation introduces slack variables that allow some points to violate the margin, penalized by a regularization parameter C. A small C tolerates more violations (wider margin, more bias); a large C penalizes violations heavily (narrower margin, more variance).
Cricket analogy: Real match situations rarely allow a perfectly clean boundary between 'safe' and 'risky' shots, so a soft-margin approach tolerates a few risky shots crossing the line, penalized by a parameter C — a low C tolerates more risky singles for a wider safety margin, a high C clamps down hard and plays it tight.
The Kernel Trick
Many datasets are not linearly separable in their original feature space. The kernel trick lets an SVM operate as if the data had been mapped into a much higher-dimensional space — where a linear separator does exist — without ever explicitly computing that mapping. A kernel function computes the dot product of two points as if they had been transformed, at a fraction of the cost. The Radial Basis Function (RBF) kernel is the most common default, controlled by a gamma parameter that governs how far the influence of a single training example reaches: small gamma gives smoother, more global boundaries, while large gamma gives tightly-fit, locally wiggly boundaries prone to overfitting.
Cricket analogy: When batting patterns aren't separable by a straight line on a wagon wheel, the kernel trick is like reprojecting the shot chart into a higher-dimensional view (say, by shot type and pace) where a clean boundary appears; the RBF kernel's gamma controls how far one shot's influence reaches, from broad smooth zones to tightly localized ones.
from sklearn.svm import SVC
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X, y = make_moons(n_samples=300, noise=0.25, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
# SVMs are distance-based, so features must be scaled first
pipe = make_pipeline(StandardScaler(), SVC(kernel='rbf'))
param_grid = {'svc__C': [0.1, 1, 10, 100], 'svc__gamma': [0.01, 0.1, 1, 'scale']}
search = GridSearchCV(pipe, param_grid, cv=5, scoring='accuracy')
search.fit(X_train, y_train)
print('Best params:', search.best_params_)
print('Test accuracy:', search.score(X_test, y_test))
print('Number of support vectors:', search.best_estimator_.named_steps['svc'].n_support_)Because only support vectors influence the boundary, SVMs can be memory-efficient at prediction time relative to methods that must consult the whole training set — but training itself scales poorly (roughly quadratic to cubic in the number of samples), which is why SVMs are rarely the first choice on datasets with hundreds of thousands of rows.
SVMs are highly sensitive to feature scale because the margin is a geometric distance. A feature measured in the thousands (e.g. income) will dominate a feature measured in single digits (e.g. age) unless both are standardized first. Always scale features before fitting an SVM.
- SVMs find the hyperplane that maximizes the margin between classes; only the closest points (support vectors) determine that boundary.
- The regularization parameter C trades off margin width against tolerance for misclassified points — small C is more regularized, large C fits training data more tightly.
- The kernel trick (e.g. RBF, polynomial) implicitly projects data into higher dimensions to make it linearly separable without the computational cost of the explicit transform.
- Gamma controls the reach of each training point's influence in RBF kernels; high gamma risks overfitting, low gamma risks underfitting.
- Feature scaling is mandatory for SVMs since the algorithm relies on geometric distances.
- SVMs work well on high-dimensional, small-to-medium datasets but scale poorly to very large training sets.
Practice what you learned
1. What defines the position of the SVM decision boundary?
2. What effect does increasing C have in a soft-margin SVM?
3. Why is the kernel trick useful?
4. What happens with a very high gamma value in an RBF kernel?
5. Why must features be scaled before fitting an SVM?
Was this page helpful?
You May Also Like
Logistic Regression Explained
Introduces logistic regression as a classification algorithm that models class probability via the sigmoid function, despite its regression-sounding name.
k-Nearest Neighbors (k-NN)
Explains the instance-based k-NN algorithm, which classifies or predicts based on the majority vote or average of the k closest training examples.
Decision Trees
Covers how decision trees split data recursively using impurity measures like Gini and entropy to produce interpretable, rule-based predictions.
Hyperparameter Tuning
Explore systematic strategies — grid search, random search, and cross-validation-based tuning — for choosing model settings that are not learned directly from data.