Anomaly Detection Basics
Anomaly detection (also called outlier detection) is the task of identifying data points that deviate significantly from the majority of the data, such as fraudulent transactions, defective sensor readings, or network intrusions. Because anomalies are by definition rare, most real-world anomaly detection problems suffer from extreme class imbalance and often lack labeled examples of every possible anomaly type, which pushes the field toward unsupervised and semi-supervised methods rather than standard supervised classification.
Cricket analogy: Spotting a timed-out dismissal is like anomaly detection; these events are so rare across thousands of matches that there aren't enough labeled examples to train a classifier, so umpires rely on general principles of normal dismissal patterns rather than pattern-matching rare cases.
Categories of Approaches
Statistical approaches flag points that fall far from the mean relative to the standard deviation (e.g. z-scores beyond 3) or outside the interquartile range (IQR) fences. Distance and density-based approaches, such as k-nearest neighbors distance or DBSCAN's noise class, flag points that are far from their neighbors or in sparse regions. Model-based approaches like Isolation Forest exploit the idea that anomalies are 'few and different', so they are easier to isolate with fewer random partitions than normal points. One-Class SVM learns a boundary around the 'normal' region of the data and flags anything outside it.
Cricket analogy: A z-score approach flags a batsman's strike rate more than three standard deviations from the team average, like Rishabh Pant's reckless innings; DBSCAN flags a fielder standing in a sparse field region as suspicious; Isolation Forest quickly isolates the squad's most unlike-everyone-else stats with just a couple of splits; One-Class SVM draws a boundary around typical Test batting and flags anyone outside it.
Supervised vs Unsupervised Anomaly Detection
When labeled anomalies exist in sufficient quantity, the problem can be framed as supervised classification, typically with heavy class-weighting or resampling (e.g. SMOTE) to counter imbalance, and evaluated with precision, recall, and the ROC/PR curves rather than raw accuracy. Far more commonly, anomalies are unlabeled or too rare and diverse to model directly, so unsupervised methods learn what 'normal' looks like from mostly-normal data and score deviations from that pattern. Semi-supervised approaches use only normal-class data during training (e.g. an autoencoder trained solely on non-fraud transactions) and flag high-reconstruction-error points at inference time as anomalies.
Cricket analogy: With enough labeled match-fixing incidents, analysts could train a weighted classifier the way selectors weight rare match-winning innings; but since fixing cases are rare, most anti-corruption units instead learn what normal betting patterns look like and flag deviations, like a semi-supervised model trained only on legitimate matches.
from sklearn.ensemble import IsolationForest
import numpy as np
# X: shape (n_samples, n_features), mostly normal observations with a few outliers
clf = IsolationForest(n_estimators=200, contamination=0.02, random_state=42)
clf.fit(X)
# -1 indicates an anomaly, 1 indicates normal
predictions = clf.predict(X)
anomaly_scores = clf.decision_function(X) # lower score = more anomalous
anomalies = X[predictions == -1]
print(f"Detected {len(anomalies)} anomalies out of {len(X)} samples")Isolation Forest works by randomly selecting a feature and a split value to partition the data recursively; anomalies, being few and different, tend to get isolated in far fewer splits than normal points, so their average path length across many random trees is much shorter.
Accuracy is a misleading metric for anomaly detection because anomalies are rare — a model that predicts 'normal' for everything can still achieve 99%+ accuracy while catching zero anomalies. Always evaluate with precision, recall, F1, or PR-AUC instead.
- Anomaly detection identifies rare data points that deviate from the majority pattern.
- Common approaches include statistical thresholds (z-score, IQR), distance/density methods, and model-based methods like Isolation Forest and One-Class SVM.
- Extreme class imbalance makes accuracy a poor evaluation metric; use precision, recall, F1, or PR-AUC instead.
- Semi-supervised methods (e.g. autoencoders trained only on normal data) flag high reconstruction error as anomalous.
- Isolation Forest isolates anomalies faster than normal points because they require fewer random splits to separate.
- The right approach depends heavily on whether labeled anomalies are available and how rare/diverse the anomalies are.
Practice what you learned
1. Why is accuracy generally a poor metric for evaluating anomaly detection models?
2. What is the core intuition behind Isolation Forest?
3. In a semi-supervised autoencoder approach to anomaly detection, what typically indicates an anomaly at inference time?
4. Which metric is most appropriate for evaluating an anomaly detector on a highly imbalanced dataset?
5. A statistical z-score-based anomaly detector flags a point as anomalous when:
Was this page helpful?
You May Also Like
DBSCAN and Density-Based Clustering
Learn how DBSCAN groups points by local density rather than distance to a centroid, letting it discover arbitrarily shaped clusters and flag outliers automatically.
k-Nearest Neighbors (k-NN)
Explains the instance-based k-NN algorithm, which classifies or predicts based on the majority vote or average of the k closest training examples.
Confusion Matrix and Classification Metrics
A structured breakdown of correct and incorrect predictions by class, forming the foundation for accuracy, precision, recall, and other classification metrics.
Precision, Recall, and F1 Score
Class-specific metrics that quantify a classifier's trustworthiness on positive predictions (precision) and its ability to find all positives (recall), balanced by the F1 score.