Support Vector Machine

Definition

A Support Vector Machine (SVM) is a supervised learning algorithm that finds the hyperplane which best separates classes of data by maximizing the margin between them, and can be extended to nonlinear boundaries using the kernel trick.

Overview

For linearly separable data, an SVM seeks the decision boundary — a hyperplane in the feature space — that maximizes the margin, the distance between the hyperplane and the nearest data points from each class, called support vectors. Maximizing this margin, rather than just finding any separating boundary, tends to produce classifiers that generalize better to unseen data, since a wider margin is more robust to small perturbations in the input. Real-world data is rarely perfectly separable, so the soft-margin SVM introduces a regularization parameter, commonly denoted C, that trades off maximizing the margin against tolerating some misclassified or margin-violating points. For data that is not linearly separable even with slack, the kernel trick allows SVMs to implicitly map inputs into a higher-dimensional feature space where a linear separator can be found, without ever explicitly computing the transformed coordinates — common kernels include the polynomial kernel and the radial basis function (RBF) kernel, which can model highly nonlinear decision boundaries. SVMs were the dominant approach for many classification tasks — text categorization, image classification, bioinformatics — through the 1990s and 2000s, prized for strong theoretical foundations (grounded in statistical learning theory) and good performance on high-dimensional data with relatively few training examples, such as gene expression data. Their popularity for tasks like image and text classification declined with the rise of deep learning, which generally scales better to very large datasets and learns feature representations automatically rather than relying on hand-chosen kernels. SVMs remain useful today for smaller or high-dimensional structured datasets, as well as for specific applications like anomaly detection (one-class SVM) and certain bioinformatics pipelines, where their sample efficiency and strong generalization guarantees are still valuable.

Key Concepts

• Finds the maximum-margin hyperplane separating classes for better generalization
• Support vectors — the closest points to the boundary — define the decision function
• Soft-margin formulation (parameter C) tolerates some misclassification for robustness
• Kernel trick enables nonlinear decision boundaries without explicit feature mapping
• Common kernels include linear, polynomial, and radial basis function (RBF)
• Strong theoretical grounding in statistical learning theory
• Performs well on high-dimensional data with relatively few samples
• One-class SVM variant used for anomaly and novelty detection

Use Cases

Frequently Asked Questions

Related in AI Fundamentals