K-Nearest Neighbors Cheat Sheet
A cheat sheet for K-Nearest Neighbors covering classification and regression in scikit-learn, distance metrics, choosing k, and scalability considerations.
1 PageBeginnerMar 18, 2026
Classifier with scikit-learn
Fit a distance-weighted KNN classifier.
python
from sklearn.neighbors import KNeighborsClassifierfrom sklearn.preprocessing import StandardScalerfrom sklearn.pipeline import make_pipelineknn = make_pipeline( StandardScaler(), KNeighborsClassifier(n_neighbors=5, weights='distance', metric='minkowski', p=2))knn.fit(X_train, y_train)print('Accuracy:', knn.score(X_test, y_test))
Choosing k
Use cross-validation to select the best neighbor count.
python
from sklearn.model_selection import cross_val_scoreimport numpy as npscores = []for k in range(1, 31, 2): # odd k avoids ties in binary classification knn = KNeighborsClassifier(n_neighbors=k) scores.append(cross_val_score(knn, X_train, y_train, cv=5).mean())best_k = list(range(1, 31, 2))[np.argmax(scores)]
KNN Regression
Predict continuous targets by averaging neighbors.
python
from sklearn.neighbors import KNeighborsRegressorreg = KNeighborsRegressor(n_neighbors=10, weights='distance')reg.fit(X_train, y_train)preds = reg.predict(X_test) # weighted average of the k nearest neighbors' targets
Key Concepts
Core theory behind KNN.
- Lazy learning- KNN has no training phase; it stores the dataset and computes distances at prediction time
- Distance metric- Euclidean (default), Manhattan, or Minkowski distance defines what 'nearest' means
- n_neighbors (k)- Small k is sensitive to noise (overfitting); large k oversmooths (underfitting)
- weights='distance'- Weights closer neighbors more heavily than farther ones when voting or averaging
- Curse of dimensionality- Distance metrics grow less meaningful as feature count increases; reduce dimensions first if needed
Querying BallTree/KDTree Directly
Bypass the estimator API to do radius queries or inspect the underlying spatial index scikit-learn builds for you.
python
from sklearn.neighbors import BallTree, KDTreeimport numpy as nptree = KDTree(X_train, leaf_size=40, metric='euclidean')# k nearest neighborsdist, idx = tree.query(X_test[:5], k=5)# radius query: all points within a fixed distance (variable neighbor count)idx_within, dist_within = tree.query_radius( X_test[:5], r=1.5, return_distance=True, sort_results=True)# BallTree generalizes better to non-Euclidean metrics (e.g. haversine)btree = BallTree(np.radians(latlon_train), metric='haversine')
Approximate Nearest Neighbors with FAISS
Trade a small amount of recall for orders-of-magnitude faster lookups on millions of vectors.
python
import faissimport numpy as npd = X_train.shape[1]quantizer = faiss.IndexFlatL2(d)index = faiss.IndexIVFFlat(quantizer, d, 100) # 100 Voronoi cellsindex.train(X_train.astype(np.float32))index.add(X_train.astype(np.float32))index.nprobe = 8 # cells searched per query; higher = more accurate, slowerdistances, neighbor_idx = index.search(X_test.astype(np.float32), k=5)
Custom & Correlation-Aware Metrics
Use Mahalanobis distance so KNN accounts for feature covariance instead of treating all axes as independent.
python
from sklearn.neighbors import KNeighborsClassifierimport numpy as npcov = np.cov(X_train, rowvar=False)VI = np.linalg.inv(cov) # inverse covariance matrixknn = KNeighborsClassifier( n_neighbors=7, metric='mahalanobis', metric_params={'VI': VI}, algorithm='brute', # tree-based algorithms don't support mahalanobis)knn.fit(X_train, y_train)
Local Outlier Factor for Anomaly Detection
Reuse KNN's neighborhood-density idea to flag points that are far sparser than their neighbors.
python
from sklearn.neighbors import LocalOutlierFactorlof = LocalOutlierFactor(n_neighbors=20, contamination=0.05, novelty=False)is_inlier = lof.fit_predict(X) # -1 = outlier, 1 = inlieranomaly_scores = lof.negative_outlier_factor_ # more negative = more anomalous# For scoring NEW points after fitting on a clean reference set:lof_novelty = LocalOutlierFactor(n_neighbors=20, novelty=True).fit(X_train_clean)new_point_labels = lof_novelty.predict(X_new)
Advanced Concepts
Complexity, indexing strategy, and preprocessing considerations that matter at scale.
- KDTree vs BallTree- KDTree splits on axis-aligned hyperplanes and degrades past ~20 dimensions; BallTree partitions with hyperspheres and handles higher dimensions and arbitrary metrics better
- leaf_size- Controls the tree/brute-force crossover point; smaller leaves speed up queries but slow down tree construction and increase memory
- Condensed Nearest Neighbor (CNN)- A prototype-selection technique that prunes the training set down to the minimal subset needed to preserve the same decision boundary, cutting prediction cost
- Gower distance- A mixed-type distance metric combining normalized numeric differences with categorical mismatch indicators, needed when features aren't all continuous
- Kernel-weighted voting- Instead of uniform or inverse-distance weights, apply a Gaussian/Epanechnikov kernel to neighbor distances for smoother decision boundaries
- Dimensionality reduction first- PCA, UMAP, or an autoencoder embedding before KNN mitigates the curse of dimensionality far more effectively than tuning k alone
- algorithm='brute' fallback- Required whenever the distance metric isn't tree-compatible (e.g. mahalanobis, cosine, or custom callables), at the cost of O(n) query time
Pro Tip
KNN's prediction cost scales with dataset size since it's a lazy learner with no training step — for large datasets, rely on sklearn's default algorithm='auto', which picks a KDTree or BallTree automatically, or use an approximate nearest-neighbor library like FAISS.
Was this cheat sheet helpful?
Explore Topics
#KNearestNeighbors#KNearestNeighborsCheatSheet#DataScience#Beginner#ClassifierWithScikitLearn#ChoosingK#KNNRegression#KeyConcepts#MachineLearning#CheatSheet#SkillVeris
