Support Vector Machines Cheat Sheet
A cheat sheet for Support Vector Machines covering kernels, margin maximization, the C and gamma hyperparameters, and scikit-learn usage.
2 PagesIntermediateFeb 25, 2026
Classifier with scikit-learn
Fit a scaled RBF-kernel SVM.
python
from sklearn.svm import SVCfrom sklearn.preprocessing import StandardScalerfrom sklearn.pipeline import make_pipelineclf = make_pipeline( StandardScaler(), SVC(kernel='rbf', C=1.0, gamma='scale', probability=True))clf.fit(X_train, y_train)print('Accuracy:', clf.score(X_test, y_test))
Kernel Comparison
Common kernel choices for SVC.
python
from sklearn.svm import SVClinear_svm = SVC(kernel='linear', C=1.0)poly_svm = SVC(kernel='poly', degree=3, C=1.0)rbf_svm = SVC(kernel='rbf', gamma=0.1, C=1.0) # default kernelsigmoid_svm = SVC(kernel='sigmoid', C=1.0)
Hyperparameter Grid Search
Tune C and gamma with cross-validation.
python
from sklearn.model_selection import GridSearchCVparam_grid = {'C': [0.1, 1, 10, 100], 'gamma': [1, 0.1, 0.01, 0.001]}grid = GridSearchCV(SVC(kernel='rbf'), param_grid, cv=5, scoring='f1')grid.fit(X_train, y_train)print(grid.best_params_)
Key Concepts
Core theory behind SVMs.
- Support vectors- Training points closest to the decision boundary; they alone define the margin
- Margin- Distance between the decision boundary and the nearest points; SVM maximizes this distance
- Kernel trick- Implicitly maps data into a higher-dimensional space to find a linear separator, without computing the mapping explicitly
- C (regularization)- Trades off margin width against classification error; a large C allows less margin violation
- gamma (RBF kernel)- Controls the influence radius of a single training point; high gamma risks overfitting
- Feature scaling- SVMs rely on distances, so features should always be standardized before fitting
Custom & Precomputed Kernels
Plug in a domain-specific similarity function instead of the built-in kernels.
python
from sklearn.svm import SVCfrom sklearn.metrics.pairwise import chi2_kernelimport numpy as np# Option 1: pass a callable kernel (computed on the fly, slower)clf = SVC(kernel=chi2_kernel)clf.fit(X_train, y_train)# Option 2: precompute the full Gram matrix yourself (fastest for repeated fits)K_train = chi2_kernel(X_train, X_train, gamma=0.5)K_test = chi2_kernel(X_test, X_train, gamma=0.5)clf_pre = SVC(kernel='precomputed')clf_pre.fit(K_train, y_train)preds = clf_pre.predict(K_test) # must be similarity to TRAIN points, in order
Support Vector Regression (SVR)
Fit a regressor that only penalizes errors outside an epsilon-insensitive tube.
python
from sklearn.svm import SVRfrom sklearn.preprocessing import StandardScalerfrom sklearn.pipeline import make_pipelinesvr = make_pipeline( StandardScaler(), SVR(kernel='rbf', C=10, epsilon=0.1, gamma='scale'))svr.fit(X_train, y_train)# epsilon widens the 'no-penalty' tube around the regression line -> larger# epsilon = fewer support vectors = sparser, smoother, less sensitive modelpreds = svr.predict(X_test)n_support_vectors = svr.named_steps['svr'].support_.shape[0]
Scaling to Large Datasets
Swap the O(n^2)-O(n^3) kernel SVM for a liblinear/SGD-based equivalent on big data.
python
from sklearn.svm import LinearSVCfrom sklearn.linear_model import SGDClassifierfrom sklearn.kernel_approximation import Nystroemfrom sklearn.pipeline import make_pipeline# LinearSVC uses liblinear (or dual=False + lbfgs) -> scales roughly linearly# in n_samples, unlike SVC's quadratic-to-cubic kernel matrix costlinear = LinearSVC(C=1.0, dual='auto', max_iter=5000)# SGDClassifier(loss='hinge') approximates a linear SVM with mini-batch# updates -> scales to millions of rows / streaming datasgd_svm = SGDClassifier(loss='hinge', alpha=1e-4, max_iter=1000)# Want a nonlinear decision boundary at scale? Approximate the RBF kernel's# feature map explicitly, then use a fast linear model on top of itkernel_approx_svm = make_pipeline( Nystroem(kernel='rbf', gamma=0.1, n_components=300), SGDClassifier(loss='hinge'))
One-Class SVM for Novelty Detection
Learn a decision boundary around 'normal' data to flag outliers with no labeled anomalies needed.
python
from sklearn.svm import OneClassSVMfrom sklearn.preprocessing import StandardScalerfrom sklearn.pipeline import make_pipeline# nu upper-bounds the fraction of training points allowed to be outliers/# support vectors -- think of it as an expected contamination rateocsvm = make_pipeline( StandardScaler(), OneClassSVM(kernel='rbf', gamma='scale', nu=0.05))ocsvm.fit(X_train_normal_only)labels = ocsvm.predict(X_test) # +1 = inlier, -1 = outlier/noveltyscores = ocsvm.named_steps['oneclasssvm'].decision_function( ocsvm.named_steps['standardscaler'].transform(X_test)) # signed distance to the boundary; more negative = more anomalous
Optimization Theory & Multi-Class Strategy
What's happening behind fit() beyond 'maximize the margin'.
- Dual formulation- SVM training solves a quadratic program over Lagrange multipliers (alpha_i) rather than the weight vector directly, which is what makes the kernel trick possible
- KKT conditions- At the optimum, alpha_i > 0 only for support vectors; non-support vectors satisfy alpha_i = 0, which is why most training points don't affect the final boundary
- Soft-margin slack variables- xi_i measures how far a point violates its margin; the primal objective balances margin width against sum(xi_i), with C controlling that trade-off
- One-vs-one (OvO) multi-class- scikit-learn's SVC trains K(K-1)/2 binary classifiers for K classes by default, which scales poorly past a few dozen classes
- One-vs-rest (OvR)- LinearSVC defaults to OvR (K classifiers total), cheaper than OvO but sensitive to class imbalance in each 'rest' bucket
- decision_function vs predict_proba- decision_function gives the raw signed distance to the boundary (fast, deterministic); predict_proba requires probability=True, which fits an extra 5-fold Platt-scaling calibration and is noticeably slower
- Training complexity- Kernel SVC training is roughly O(n_samples^2) to O(n_samples^3) depending on the solver, which is why it becomes impractical well before a million rows
Pro Tip
Always scale features before training an SVM — because the algorithm depends on distances and dot products, an unscaled feature with a large numeric range will dominate the margin calculation and quietly degrade model performance.
Was this cheat sheet helpful?
Explore Topics
#SupportVectorMachines#SupportVectorMachinesCheatSheet#DataScience#Intermediate#ClassifierWithScikitLearn#KernelComparison#HyperparameterGridSearch#KeyConcepts#MachineLearning#CheatSheet#SkillVeris
