Removing Duplicates
Duplicate rows creep into datasets from repeated data entry, double-submitted forms, or careless joins/concatenations that fan out matching rows. Left unaddressed, duplicates silently inflate counts, sums, and averages, and can bias any downstream model trained on the data. pandas provides two closely related tools for dealing with them: duplicated(), which flags duplicate rows as a boolean Series for inspection, and drop_duplicates(), which removes them outright.
Cricket analogy: When a scorer at a local T20 league re-enters the same boundary twice because two volunteers submitted the same ball, the run total is inflated; duplicated() flags such repeated entries and drop_duplicates() removes the extra copy so Virat Kohli's tally stays accurate.
Detecting duplicates with duplicated()
df.duplicated() returns True for every row that is an exact repeat of an earlier row (by default, comparing all columns), and False for the first occurrence of each unique row. The keep parameter controls which occurrence is treated as 'not a duplicate': keep='first' (the default) marks later occurrences as duplicates, keep='last' marks earlier ones, and keep=False marks every occurrence of any repeated row as a duplicate, useful when you want to inspect or remove all copies rather than retain one.
Cricket analogy: df.duplicated() marks Rohit Sharma's second logged century as a duplicate of the first identical scorecard row, while keep='last' would instead flag the earlier entry, and keep=False flags every copy so a scorer can review all repeats before deciding what to trust.
import pandas as pd
logins = pd.DataFrame({
'user_id': [101, 102, 101, 103, 102],
'device': ['mobile', 'web', 'mobile', 'web', 'web'],
'session_id': ['a1', 'b1', 'a1', 'c1', 'b2'],
})
print(logins.duplicated())
# 0 False
# 1 False
# 2 True <- exact repeat of row 0
# 3 False
# 4 False
# dtype: bool
# Deduplicate on a SUBSET of columns: same user+device counted once
deduped = logins.drop_duplicates(subset=['user_id', 'device'], keep='first')
print(deduped)
# user_id device session_id
# 0 101 mobile a1
# 1 102 web b1
# 3 103 web c1
# See every row that has at least one duplicate (by user_id+device), all copies
all_dupes = logins[logins.duplicated(subset=['user_id', 'device'], keep=False)]Deduplicating with drop_duplicates()
drop_duplicates() mirrors duplicated()'s subset and keep arguments but returns a filtered DataFrame instead of a boolean mask. Restricting subset to a business key (like user_id + device, or an email address) rather than comparing every column is often the correct choice, since two rows can be 'the same event' for deduplication purposes even if a timestamp or log message differs slightly between them.
Cricket analogy: drop_duplicates(subset=['player_id','match_id']) treats two rows for Rohit Sharma in the same match as the same event even if the ball-by-ball commentary text differs slightly, keeping deduplication focused on the meaningful business key rather than every column.
Deciding what counts as a 'duplicate' is a modeling decision, not just a technical one: two purchase records with identical customer_id, product_id, and amount might be a genuine repeat order rather than a data entry error. Always deduplicate on a deliberately chosen subset of columns that reflects what uniquely identifies a real-world entity or event in your domain.
Calling drop_duplicates() with no subset argument compares every column, so two rows that differ only in an auto-generated timestamp or ID column will never be flagged as duplicates even though they represent the same underlying event. Always think about whether the full-row comparison actually matches your definition of 'duplicate' before relying on the default behavior.
- duplicated() returns a boolean mask flagging duplicate rows; drop_duplicates() returns a DataFrame with duplicates removed.
- By default, both compare all columns and keep the first occurrence of each unique row.
- The subset parameter restricts comparison to specific columns, matching your real-world definition of a duplicate entity/event.
- keep='first', 'last', or False controls which occurrence(s) are retained or flagged.
- Full-row comparison misses duplicates that differ only in irrelevant columns like auto-generated timestamps or IDs.
- What counts as a duplicate is a domain decision — choose the subset of columns deliberately rather than defaulting blindly.
Practice what you learned
1. What does df.duplicated(keep='first') mark as True?
2. What is the effect of passing keep=False to drop_duplicates()?
3. Why might you use the subset parameter with drop_duplicates() instead of comparing all columns?
4. If two rows are identical except for an auto-incrementing log ID column, what does df.drop_duplicates() (no subset) do?
5. Which expression returns all rows that have at least one duplicate, including every copy, based only on 'email' and 'signup_date'?
Was this page helpful?
You May Also Like
Handling Missing Data in Pandas
Learn how pandas represents missing values with NaN and NA, and the core toolkit — isna, dropna, fillna, interpolate — for detecting and handling them.
Boolean Filtering
Learn how to filter DataFrame rows using boolean masks built from comparison and logical operators, including combining conditions and using isin and query for cleaner syntax.
Handling Outliers
Learn common statistical techniques — z-score, IQR, and percentile capping — for detecting and treating outliers in pandas and NumPy data.
Data Type Conversion
Master how to inspect and convert pandas column dtypes with astype, to_numeric, to_datetime, and category types to fix incorrect or inefficient typing.