How do you handle missing data in a dataset?
Learn to handle missing data: diagnose MCAR, MAR, MNAR, choose deletion or imputation like median and MICE, and avoid data leakage.
Expected Interview Answer
You handle missing data by first understanding why it is missing, then choosing between deletion, imputation, or model-based methods, guided by how much is missing and the missingness mechanism (MCAR, MAR, or MNAR).
Start by quantifying and visualizing the gaps and diagnosing the mechanism: missing completely at random, missing at random, or missing not at random. For small, random gaps you may drop rows or columns; more often you impute using mean, median, mode, forward-fill, k-NN, or model-based methods like MICE. Adding a missingness indicator can preserve signal, and imputation should be fit on training data only to avoid leakage.
- Prevents biased or crashing models
- Preserves valuable rows instead of discarding them
- Chooses the method to fit the missingness mechanism
- Avoids data leakage by fitting on training data only
- Retains signal via missingness indicators
AI Mentor Explanation
A scorebook with a few deliveries smudged out. If only a couple of balls are illegible you can drop them, but if a whole over is gone you estimate runs from the bowler's usual economy and the batter's rate rather than tossing the innings. How you fill the gaps must match why they went missing, a rain-break gap differs from a lazy scorer skipping entries.
Step-by-Step Explanation
Step 1
Quantify the gaps
Measure the percentage missing per column and visualize patterns with a missingness matrix.
Step 2
Diagnose the mechanism
Decide whether data is MCAR, MAR, or MNAR to choose an unbiased strategy.
Step 3
Decide delete vs impute
Drop rows/columns only when loss is small and random; otherwise impute.
Step 4
Choose an imputation method
Use mean/median/mode, forward-fill, k-NN, or MICE depending on data type and structure.
Step 5
Fit on train only
Learn imputation parameters from the training set and apply them to validation and test to avoid leakage.
Step 6
Flag missingness
Optionally add a binary indicator column so the model can learn from the fact a value was missing.
What Interviewer Expects
- Awareness of MCAR, MAR, and MNAR
- Trade-offs between deletion and imputation
- Multiple imputation techniques named
- Awareness of data leakage during imputation
- Use of missingness indicators when appropriate
Common Mistakes
- Always dropping rows regardless of how much is lost
- Imputing before the train-test split, causing leakage
- Using mean imputation for skewed data instead of median
- Ignoring why the data is missing
- Filling categorical columns with a numeric average
Best Answer (HR Friendly)
“First I check how much data is missing and why. If it's a tiny random amount I might drop those rows, but usually I fill the gaps with a sensible estimate like the median or a predicted value, making sure I only learn that estimate from the training data.”
Code Example
import pandas as pd
from sklearn.impute import SimpleImputer
# See how much is missing per column
print(df.isna().mean().sort_values(ascending=False))
# Median imputation, fit on training data only
imputer = SimpleImputer(strategy='median')
X_train_imputed = imputer.fit_transform(X_train)
X_test_imputed = imputer.transform(X_test)
# Preserve the signal that a value was missing
df['income_missing'] = df['income'].isna().astype(int)Follow-up Questions
- What is the difference between MCAR, MAR, and MNAR?
- When would you prefer median over mean imputation?
- How does MICE (multiple imputation) work?
- Why is imputing before the train-test split a problem?
- How do you impute missing categorical values?
MCQ Practice
1. Why should imputation parameters be fit on the training set only?
Fitting on all data leaks test information into the imputer, giving an over-optimistic evaluation.
2. For a heavily skewed numeric column, which simple imputation is safest?
The median is robust to skew and outliers, whereas the mean is pulled toward extreme values.
3. Data missing depending on the unobserved value itself is called?
MNAR (missing not at random) means the probability of missingness depends on the missing value, which is the hardest case.
Flash Cards
First step with missing data? — Quantify how much is missing and diagnose the mechanism (MCAR, MAR, MNAR).
When is deletion acceptable? — When the missing amount is small and missing completely at random.
Why fit imputation on training data only? — To prevent leakage of test information into the model.
What is a missingness indicator? — A binary column flagging that a value was missing, so the model can learn from it.