Data Cleaning & Preprocessing Cheat Sheet
Practical pandas workflows for handling missing values, duplicates, outliers, and inconsistent data types before modeling.
2 PagesBeginnerMar 18, 2026
Missing Values & Duplicates
Detect, impute, and deduplicate rows.
python
import pandas as pddf = pd.read_csv("data.csv")# Inspect missingnessprint(df.isnull().sum())print(df.isnull().mean() * 100) # % missing per column# Drop rows/columns with too many missing valuesdf = df.dropna(thresh=len(df.columns) * 0.7) # keep rows with >=70% non-nulldf = df.drop(columns=["mostly_empty_col"])# Impute missing valuesdf["age"] = df["age"].fillna(df["age"].median())df["city"] = df["city"].fillna(df["city"].mode()[0])df["income"] = df.groupby("region")["income"].transform(lambda x: x.fillna(x.mean()))# Duplicatesprint(df.duplicated().sum())df = df.drop_duplicates(subset=["user_id"], keep="last")
Outliers & Data Types
Detect outliers and fix inconsistent columns.
python
# Detect outliers with the IQR methodQ1, Q3 = df["income"].quantile([0.25, 0.75])IQR = Q3 - Q1lower, upper = Q1 - 1.5 * IQR, Q3 + 1.5 * IQRoutliers = df[(df["income"] < lower) | (df["income"] > upper)]# Cap (winsorize) instead of droppingdf["income"] = df["income"].clip(lower, upper)# Z-score methodz_scores = (df["income"] - df["income"].mean()) / df["income"].std()df = df[z_scores.abs() < 3]# Fix dtypes and inconsistent stringsdf["date"] = pd.to_datetime(df["date"], errors="coerce")df["price"] = pd.to_numeric(df["price"].str.replace("$", ""), errors="coerce")df["category"] = df["category"].str.strip().str.lower()
Cleaning Checklist
Standard steps to run on any new dataset.
- Missing values- check with isnull().sum(); impute (mean/median/mode) or drop based on missingness %
- Duplicates- detect with duplicated(), remove with drop_duplicates()
- Outliers- detect via IQR or z-score; decide to cap, transform, or remove based on domain knowledge
- Inconsistent types- coerce columns to the correct dtype with pd.to_numeric/pd.to_datetime
- Inconsistent categories- normalize casing/whitespace (e.g. 'NY' vs 'ny ' vs 'New York')
- Structural errors- fix typos, inconsistent units, or mislabeled columns
- Leakage columns- drop features that wouldn't be available at prediction time
- Class imbalance- check the target distribution before modeling; consider resampling or class weights
Imputation Strategies
How to fill in missing values responsibly.
- Mean/median imputation- simple and fast; median is more robust to skew and outliers
- Mode imputation- standard choice for categorical columns
- Group-wise imputation- fill using the mean/median within a related group (e.g. by region)
- Forward/backward fill- propagate the last/next known value; common for time series
- Model-based imputation- predict missing values from other features (e.g. KNNImputer, IterativeImputer)
- Missing indicator column- add a binary flag for 'was this value missing' to preserve that signal
Leak-Proof Pipelines with ColumnTransformer
Bundle imputation, scaling, and encoding into one fitted object so preprocessing statistics never leak across train/test splits.
python
from sklearn.compose import ColumnTransformerfrom sklearn.pipeline import Pipelinefrom sklearn.impute import SimpleImputerfrom sklearn.preprocessing import StandardScaler, OneHotEncodernum_cols = ["age", "income"]cat_cols = ["city", "category"]num_pipe = Pipeline([ ("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler()),])cat_pipe = Pipeline([ ("impute", SimpleImputer(strategy="most_frequent")), ("encode", OneHotEncoder(handle_unknown="ignore")),])preprocess = ColumnTransformer([ ("num", num_pipe, num_cols), ("cat", cat_pipe, cat_cols),])# fit only on train, transform both -- no statistics leak into testX_train_t = preprocess.fit_transform(X_train)X_test_t = preprocess.transform(X_test)
Multivariate Outlier Detection
IQR/z-score only look at one column at a time -- these catch outliers defined by unusual combinations of features.
python
from sklearn.ensemble import IsolationForestfrom sklearn.neighbors import LocalOutlierFactorfrom scipy.spatial import distanceimport numpy as np# Isolation Forest: isolates anomalies via random partitioningiso = IsolationForest(contamination=0.02, random_state=42)df["is_outlier_iso"] = iso.fit_predict(df[num_cols]) == -1# Local Outlier Factor: flags points with much lower density than neighborslof = LocalOutlierFactor(n_neighbors=20, contamination=0.02)df["is_outlier_lof"] = lof.fit_predict(df[num_cols]) == -1# Mahalanobis distance: accounts for correlation between featurescov = np.cov(df[num_cols].values, rowvar=False)inv_cov = np.linalg.inv(cov)mean = df[num_cols].mean().valuesdf["mahalanobis"] = df[num_cols].apply( lambda row: distance.mahalanobis(row.values, mean, inv_cov), axis=1)
Text Normalization & Fuzzy Deduplication
Clean free-text columns and catch near-duplicate records that exact matching misses.
python
import refrom rapidfuzz import fuzz# Normalize free-text columnsdf["name"] = ( df["name"] .str.strip() .str.lower() .str.replace(r"[^a-z0-9\s]", "", regex=True) .str.replace(r"\s+", " ", regex=True))# Extract structured values with regexdf["zip_code"] = df["address"].str.extract(r"(\d{5})(?:-\d{4})?$")# Fuzzy-match near-duplicate names (e.g. "jon smith" vs "john smith")def is_near_duplicate(a, b, threshold=90): return fuzz.token_sort_ratio(a, b) >= thresholdflags = [ is_near_duplicate(df["name"].iloc[i], df["name"].iloc[i - 1]) for i in range(1, len(df))]
Categorical Encoding Strategies
Choosing the right encoder matters as much as choosing the right imputer.
- One-hot encoding- best for low-cardinality nominal columns; explodes width with high cardinality
- Ordinal encoding- for genuinely ordered categories (e.g. 'low' < 'medium' < 'high')
- Target/mean encoding- replaces a category with the mean target for that category; must be cross-fitted to avoid leakage
- Frequency encoding- replaces a category with its occurrence count/rate; cheap and leakage-free
- Hashing trick- fixed-width hash of category strings; handles unseen categories and huge cardinality at the cost of collisions
- Rare-category bucketing- group categories below a frequency threshold into an 'Other' bucket before encoding
- WoE (Weight of Evidence)- log-odds transform per category, common in credit scoring for logistic models
Memory Optimization via Dtype Downcasting
Shrink a DataFrame's memory footprint before scaling to large datasets.
python
import pandas as pddef optimize_dtypes(df): for col in df.select_dtypes(include="int64").columns: df[col] = pd.to_numeric(df[col], downcast="integer") for col in df.select_dtypes(include="float64").columns: df[col] = pd.to_numeric(df[col], downcast="float") for col in df.select_dtypes(include="object").columns: if df[col].nunique() / len(df) < 0.5: # low-cardinality -> category dtype df[col] = df[col].astype("category") return dfbefore = df.memory_usage(deep=True).sum() / 1e6df = optimize_dtypes(df)after = df.memory_usage(deep=True).sum() / 1e6print(f"{before:.1f} MB -> {after:.1f} MB")
Pro Tip
Never impute missing values or drop outliers before splitting into train/test sets — compute imputation statistics (median, mean) only on the training set, then apply them to test data, or you'll leak test-set information into training.
Was this cheat sheet helpful?
Explore Topics
#DataCleaningPreprocessing#DataCleaningPreprocessingCheatSheet#DataScience#Beginner#MissingValuesDuplicates#OutliersDataTypes#CleaningChecklist#ImputationStrategies#MachineLearning#CheatSheet#SkillVeris
