Duplicate Detection and Data Consistency — Data Analysis & Feature Engineering

Duplicates and inconsistencies are the quiet data-quality problems that inflate counts, bias models, and corrupt joins, and they require deliberate detection because they often hide behind superficial differences. Handling them exists as a discipline because real data, assembled from multiple sources and entered by many hands, accumulates exact duplicate rows, near-duplicates that differ only in formatting, and inconsistent representations of the same entity — 'CSK', 'Chennai Super Kings', and 'chennai super kings' all meaning one team. Left undetected, duplicates overweight repeated records and leak between train and test sets, while inconsistencies fragment a single category into several, breaking aggregations and encodings. The techniques — exact and fuzzy duplicate detection, and consistency normalisation through standardising formats and resolving entities — are what transform messy, redundant data into a clean, coherent dataset that downstream analysis can trust.

Analogy🏏Cricket

🏏 Think of it like cricket: Reducing twenty batting statistics to two dimensions with PCA produces a flat projection that may look like one undifferentiated cloud, while t-SNE or UMAP finds that the data actually organises into distinct clusters — aggressive pinch-hitters, steady anchors, explosive finishers — whose separation PCA's flat projection smeared together. Just as the non-linear reduction reveals the natural groupings that flat projection could not, t-SNE and UMAP reveal structure that PCA misses because it can only flatten, not curve to follow the data's natural shape. The insight is that non-linear reduction methods follow the data's true curved geometry rather than forcing a flat projection, revealing the cluster structure and local neighbourhoods that linear methods like PCA cannot preserve.