Data Cleaning Basics
Real-world data is rarely analysis-ready. It arrives with duplicate records, inconsistent formatting, incorrect data types, obvious data-entry errors, and outliers that may or may not be legitimate. Data cleaning is the process of detecting and correcting these problems before features are engineered and a model is trained. The old programming adage 'garbage in, garbage out' applies with particular force to ML: a model trained on dirty data will confidently learn the noise and errors right along with the genuine signal.
Cricket analogy: Raw scorecard data arrives full of duplicate entries, inconsistent player-name spellings, and clearly wrong figures like a 150-not-out logged as a wicket; cleaning this before building a player-rating model matters because a model trained on the errors will confidently learn the typos alongside genuine batting signal.
Duplicates and Inconsistent Formatting
Duplicate rows — whether exact copies or near-duplicates caused by repeated data entry — inflate the apparent importance of those records and can bias a model toward whatever pattern the duplicates represent. Inconsistent formatting is equally common: the same category might appear as 'NY', 'New York', and 'new york' across different rows due to manual entry or merged data sources. Left uncorrected, a model (or even a simple groupby) will treat these as entirely different categories, fragmenting what should be a single group and weakening any pattern involving it.
Cricket analogy: Duplicate scorecard entries for the same innings inflate that innings' apparent weight in a batting-average calculation, biasing conclusions toward the duplicate; and inconsistent team naming like RCB, Royal Challengers Bangalore, and royal challengers bengaluru fragments what should be one team's stats into three weaker-looking groups.
Type Errors and Outliers
Columns are sometimes stored as the wrong data type — a numeric column read in as text because of a stray currency symbol or comma, or a date stored as a free-text string in inconsistent formats. These need explicit conversion and validation. Outliers — values far outside the typical range — require careful judgment rather than blanket removal: some outliers are data-entry errors (an age of 200) that should be fixed or dropped, while others are genuine, informative extreme values (a legitimate very-high-income customer) that carry real signal and should usually be kept, perhaps after using a robust model or transformation to reduce their influence.
Cricket analogy: A strike rate stored as text because of a stray percent symbol needs explicit conversion before analysis; and an outlier like a 500-not-out score needs judgment — likely a data-entry error to fix, whereas a genuinely rare high score like Brian Lara's 400* is real, informative signal that should be kept, perhaps down-weighted.
import pandas as pd
df = pd.read_csv('raw_customers.csv')
# 1. Remove exact duplicate rows
df = df.drop_duplicates()
# 2. Standardize inconsistent categorical formatting
df['state'] = df['state'].str.strip().str.upper()
df['state'] = df['state'].replace({'NEW YORK': 'NY', 'CALIFORNIA': 'CA'})
# 3. Fix a numeric column stored as text with stray characters
df['income'] = (
df['income']
.astype(str)
.str.replace('[$,]', '', regex=True)
.astype(float)
)
# 4. Flag (not blindly drop) extreme outliers using the IQR method
Q1, Q3 = df['income'].quantile([0.25, 0.75])
IQR = Q3 - Q1
lower, upper = Q1 - 1.5 * IQR, Q3 + 1.5 * IQR
df['income_is_outlier'] = ~df['income'].between(lower, upper)
print(f"Rows after dedup: {len(df)}, flagged outliers: {df['income_is_outlier'].sum()}")A practical workflow tip: always cross-check row counts, unique category values (df['col'].unique()), and summary statistics (df.describe()) before and after each cleaning step, so you can quantify exactly how much data each transformation affected — and catch cleaning steps that accidentally destroy far more data than intended.
Never clean the test set differently from how you clean the training set, and never let cleaning decisions be informed by looking at test-set values (e.g. picking an outlier threshold based on the full dataset). Doing so introduces a subtle form of data leakage and biases your evaluation.
- Data cleaning fixes duplicates, inconsistent formatting, wrong data types, and outliers before modeling begins.
- Duplicate rows can bias a model by over-weighting whatever pattern the duplicated records represent.
- Inconsistent categorical formatting (e.g. 'NY' vs 'New York') fragments what should be a single category.
- Outliers require judgment: some are entry errors to fix or remove, others are genuine, informative extreme values.
- Cleaning steps should be defined on the training set and applied consistently, never tuned by peeking at test data.
- Systematically comparing row counts and summary statistics before/after cleaning helps catch unintended data loss.
Practice what you learned
1. Why are duplicate rows problematic for model training?
2. What problem does inconsistent categorical formatting (e.g. 'NY' vs 'New York') cause?
3. According to the article, how should outliers generally be handled?
4. Why is it problematic to choose a cleaning threshold (like an outlier cutoff) based on statistics computed from the full dataset including the test set?
5. In the code example, what method is used to flag potential income outliers?
Was this page helpful?
You May Also Like
Handling Missing Data
Explores why data goes missing, the different missingness mechanisms, and practical strategies — from deletion to imputation — for preparing incomplete datasets for modeling.
Datasets, Features, and Labels
Explains the vocabulary and structure of ML data — datasets, samples, features, and labels — and how they are organized before a model can be trained.
Feature Scaling and Normalization
Learn why rescaling numeric features onto comparable ranges matters, and how standardization, min-max scaling, and robust scaling each affect model behavior.
The Machine Learning Workflow
An end-to-end walkthrough of the standard machine learning project lifecycle, from problem framing and data collection through model deployment and monitoring.