Outlier Detection — IQR, Z-Score, Isolation Forest — Data Analysis & Feature Engineering

Outliers are observations that deviate markedly from the rest of the data, and detecting them is a critical EDA task because they can be either valuable signals or corrupting errors, and the two demand opposite responses. Outlier detection exists because extreme values exert disproportionate influence on means, variances, correlations, and models, so leaving them undetected lets a handful of points distort entire analyses, while removing genuine rare events destroys real information. The challenge is that outliers come in different forms and contexts, and no single method catches them all: statistical methods like the IQR fence and Z-score work for single variables, while multivariate methods like the Isolation Forest catch points that are unremarkable on each variable alone but anomalous in combination. Mastering a toolkit of detection methods, and the judgement of how to respond, is essential to trustworthy analysis.

Analogy🏏Cricket

🏏 Think of it like cricket: Reducing twenty batting statistics to two dimensions with PCA produces a flat projection that may look like one undifferentiated cloud, while t-SNE or UMAP finds that the data actually organises into distinct clusters — aggressive pinch-hitters, steady anchors, explosive finishers — whose separation PCA's flat projection smeared together. Just as the non-linear reduction reveals the natural groupings that flat projection could not, t-SNE and UMAP reveal structure that PCA misses because it can only flatten, not curve to follow the data's natural shape. The insight is that non-linear reduction methods follow the data's true curved geometry rather than forcing a flat projection, revealing the cluster structure and local neighbourhoods that linear methods like PCA cannot preserve.