Missing Data — Mechanisms and Strategies — Data Analysis & Feature Engineering

Missing data is the near-universal reality of real datasets, and how you handle it can make or break an analysis, because the wrong strategy introduces bias as surely as the missingness itself. Handling missing data exists as a discipline because most algorithms cannot operate on absent values, yet naively deleting or filling them can distort distributions, fabricate relationships, or systematically skew results. The crucial insight is that not all missingness is equal — values can be missing completely at random, missing at random conditional on other variables, or missing not at random in ways tied to the unseen value itself — and the right response depends entirely on which mechanism is at work. Mastering the mechanisms and the matching strategies is what separates principled missing-data handling from the careless deletion and mean-filling that quietly corrupt countless analyses.

Analogy🏏Cricket

🏏 Think of it like cricket: Reducing twenty batting statistics to two dimensions with PCA produces a flat projection that may look like one undifferentiated cloud, while t-SNE or UMAP finds that the data actually organises into distinct clusters — aggressive pinch-hitters, steady anchors, explosive finishers — whose separation PCA's flat projection smeared together. Just as the non-linear reduction reveals the natural groupings that flat projection could not, t-SNE and UMAP reveal structure that PCA misses because it can only flatten, not curve to follow the data's natural shape. The insight is that non-linear reduction methods follow the data's true curved geometry rather than forcing a flat projection, revealing the cluster structure and local neighbourhoods that linear methods like PCA cannot preserve.