Handling Missing Data
Missing values are one of the most common problems in real-world datasets: a sensor fails to report a reading, a survey respondent skips a question, or two data sources merge imperfectly and leave gaps. Most ML algorithms cannot handle missing values directly and will raise an error or produce nonsensical results if fed a matrix containing NaNs. Deciding how to handle missing data is not just a technical formality — the chosen strategy can materially change what the model learns, so it deserves careful thought rather than a reflexive default.
Cricket analogy: When a scorer's tablet glitches and a ball's outcome goes unrecorded, the official scorecard can't just skip it silently — someone must decide how to fill that gap, since guessing wrong could misstate a player's average.
Why Data Is Missing: MCAR, MAR, and MNAR
Statisticians distinguish three mechanisms behind missing data. Missing Completely at Random (MCAR) means the chance of a value being missing has nothing to do with any variable, observed or not — for example, a random sensor glitch. Missing at Random (MAR) means the missingness depends on other observed variables — for instance, older respondents in a survey being less likely to report their income, but conditional on age, missingness is unrelated to the income value itself. Missing Not at Random (MNAR) is the trickiest case: the missingness depends on the unobserved value itself, such as very high earners being less likely to report their income precisely because it is high. The right handling strategy depends heavily on which mechanism is at play, and MNAR situations are especially dangerous because naive imputation can systematically bias the model.
Cricket analogy: A stadium's ball-tracking camera randomly glitching on any delivery is MCAR; a scorer more often missing deliveries late in a rain-shortened match (observable: overs remaining) is MAR; but a bowler's team quietly not reporting a no-ball because it favors them is MNAR, since the missingness depends on the unrecorded outcome itself.
Strategies: Deletion and Imputation
The simplest strategy is deletion: dropping rows with missing values (listwise deletion) or dropping entire columns that are mostly missing. This is safe and simple when missingness is rare and roughly MCAR, but it discards information and can introduce bias if missingness is not random, and it becomes wasteful when many different columns each have a few scattered missing values, since deletion could eliminate most of the dataset. Imputation instead fills in missing values with a plausible substitute: a constant, the column mean or median (for numerical features), the mode (for categorical features), a value predicted from other features (model-based imputation like KNNImputer), or a special 'missing' category. Adding a companion boolean 'was_missing' indicator column alongside an imputed feature is a common technique that preserves the information that a value was originally missing, which can itself be predictive.
Cricket analogy: Dropping every match where a player's strike rate wasn't recorded is simple but wastes a whole season of data if many matches have small gaps; filling gaps with the player's season average, plus flagging which entries were estimated, preserves more of the record.
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer, KNNImputer
df = pd.DataFrame({
'age': [25, np.nan, 42, 31, np.nan],
'income': [50000, 62000, np.nan, 71000, 48000],
})
# Preserve the missingness signal before imputing
df['age_was_missing'] = df['age'].isna()
# Median imputation is robust to outliers for numerical features
imputer = SimpleImputer(strategy='median')
df[['age', 'income']] = imputer.fit_transform(df[['age', 'income']])
# Alternative: model-based imputation using similarity between rows
knn_imputer = KNNImputer(n_neighbors=3)
df_knn = pd.DataFrame(
knn_imputer.fit_transform(df[['age', 'income']]),
columns=['age', 'income']
)
print(df)Always fit imputers (like SimpleImputer or KNNImputer) on the training set only, then apply the same fitted transform to the validation and test sets — exactly like scalers and encoders — to avoid leaking information about the held-out data's distribution into training.
Imputing missing values with the mean or median can artificially shrink variance and weaken correlations, especially when a large fraction of values are missing. Always check what percentage of each column is missing, and consider dropping columns that are missing above roughly 40-50% of the time, since heavy imputation there mostly manufactures signal rather than preserving it.
- Most ML algorithms cannot handle raw NaN values and require an explicit missing-data strategy before training.
- Missingness mechanisms — MCAR, MAR, and MNAR — determine how safe simple approaches like deletion or mean imputation are.
- Deletion (row or column) is simple but discards information and can bias results if missingness isn't random.
- Imputation fills gaps with a constant, statistic (mean/median/mode), or model-predicted value like KNNImputer.
- Adding a 'was_missing' indicator column preserves potentially predictive information about the missingness itself.
- Imputers must be fit only on training data and applied consistently to validation/test data to avoid leakage.
Practice what you learned
1. What does MCAR (Missing Completely at Random) mean?
2. Why is MNAR (Missing Not at Random) considered especially dangerous?
3. What is a key drawback of listwise deletion when many columns each have a few missing values?
4. What is the purpose of adding a 'was_missing' indicator column?
5. Why should an imputer be fit only on the training set, not the full dataset?
Was this page helpful?
You May Also Like
Data Cleaning Basics
Covers the core techniques for detecting and fixing messy real-world data — duplicates, inconsistent formatting, outliers, and type errors — before it reaches a model.
Feature Scaling and Normalization
Learn why rescaling numeric features onto comparable ranges matters, and how standardization, min-max scaling, and robust scaling each affect model behavior.
Datasets, Features, and Labels
Explains the vocabulary and structure of ML data — datasets, samples, features, and labels — and how they are organized before a model can be trained.
What Is Feature Engineering?
Explore how transforming raw data into informative input variables often matters more for model quality than the choice of algorithm itself.