Handling Duplicates and Outliers in Data

1How Do You Handle Duplicates and Outliers?

Handle duplicates by detecting them with df.duplicated() and removing them with df.drop_duplicates(), and handle outliers by flagging extreme values with the IQR or z-score method and then deciding whether to keep, cap, or drop them. The golden rule is to investigate before you delete — both duplicates and outliers can be legitimate.

Duplicates usually come from data-entry errors or joins gone wrong, while outliers are simply values far from the rest. Neither is automatically bad, so cleaning is as much about judgement as it is about code.

2Finding Duplicate Rows

df.duplicated() returns a boolean Series marking each row that is a repeat of an earlier one. By default a row counts as a duplicate only if every column matches, but subset= lets you define duplication on the columns that actually identify a record, such as an email or order ID.

• df.duplicated().sum() # how many duplicate rows exist
• df[df.duplicated(keep=False)] # show all copies, not just the extras
• df.duplicated(subset=['email']) # duplicates by a business key
• df['id'].value_counts()[lambda s: s > 1] # which ids repeat

3Removing Duplicates

drop_duplicates() keeps one copy of each duplicated row and discards the rest. The keep argument decides which copy survives — the first occurrence, the last, or none at all. Choosing last is common when later records are more up to date.

• df.drop_duplicates() # keep the first of each duplicate
• df.drop_duplicates(subset=['email'], keep='last') # newest wins
• df.drop_duplicates(keep=False) # drop every row that has any duplicate
• df = df.sort_values('updated_at').drop_duplicates('id', keep='last')

4Detecting Outliers

An outlier is a value that sits far from the bulk of the data. Two classic detection methods cover most cases. The IQR rule flags values below Q1 minus 1.5 times the interquartile range or above Q3 plus 1.5 times it. The z-score method flags values more than roughly three standard deviations from the mean.

• q1, q3 = df['x'].quantile([0.25, 0.75])
• iqr = q3 - q1
• mask = (df['x'] < q1 - 1.5 * iqr) | (df['x'] > q3 + 1.5 * iqr)
• z = (df['x'] - df['x'].mean()) / df['x'].std() # z-score alternative
• outliers = df[z.abs() > 3]

5Treating Outliers

Once flagged, an outlier has three possible fates: keep it, cap it, or drop it. Capping — replacing extreme values with a boundary, also called winsorising — often beats deletion because it tames the value's influence without discarding the row entirely.

• Keep: the extreme is real and meaningful (a genuine high-value customer).
• Cap: clip to the IQR bounds with df['x'].clip(lower, upper) to reduce leverage.
• Drop: remove only when you are confident it is a data-entry error.
• Transform: a log transform can compress a long right tail so extremes matter less.

6Common Mistakes to Avoid

Cleaning too aggressively causes as many problems as not cleaning at all.

• Deleting duplicates without checking subset= and losing legitimately distinct rows.
• Removing outliers before understanding why they exist — you may be deleting your most interesting cases.
• Applying the z-score method to heavily skewed data, where it flags the wrong points.
• Forgetting that drop_duplicates depends on row order — sort first.
• Cleaning on the full dataset before splitting, which can leak information into a test set.

7A Practical Cleaning Workflow

Cleaning is most reliable when it follows a repeatable order rather than ad-hoc fixes. Duplicates first, then outliers, keeps each step from confusing the next — deduplicating after outlier removal, for instance, can leave you re-examining rows you already dropped.

• Profile the data with describe() and value_counts() to see what you are dealing with.
• Detect and resolve duplicates using a business-key subset.
• Flag outliers with the IQR or z-score method and inspect them individually.
• Decide keep, cap, or drop per outlier, and record the rationale.
• Re-profile afterwards to confirm the distribution looks sensible.

8Key Takeaways

Clean deliberately, not reflexively.

• Find duplicates with duplicated(); remove with drop_duplicates() and a sensible subset.
• keep='first'/'last'/False controls which copies survive, and order matters.
• Detect outliers with the IQR rule (robust) or z-score (assumes normality).
• Prefer capping or transforming over deleting real extremes.
• Investigate and log every removal — outliers are sometimes the signal.

9Frequently Asked Questions

Q: Should I always remove outliers from my data? A: No. An outlier is only a problem if it is an error or if it distorts the specific analysis you are doing. In domains like fraud detection or anomaly monitoring, outliers are the target. Investigate the cause first, then decide whether to keep, cap, or drop.

Q: What is the difference between the IQR and z-score methods? A: The IQR method flags values outside Q1 minus 1.5 times the interquartile range and Q3 plus 1.5 times it, and is robust to skew. The z-score method flags values more than about three standard deviations from the mean and assumes a roughly normal distribution, so it is less reliable on skewed data.

Q: How do I remove duplicate rows but keep the most recent one? A: Sort the frame by a timestamp, then call drop_duplicates with keep='last', for example df.sort_values('updated_at').drop_duplicates('id', keep='last'). Sorting first makes the choice of which row to keep deterministic and intentional.

Q: What does winsorising mean? A: Winsorising, or capping, replaces extreme values with a chosen boundary — for example clipping everything above the 99th percentile down to that percentile. It reduces the influence of outliers on statistics and models while keeping the rows in the dataset.