Transform and Filter on Groups
Aggregation is not the only operation groupby supports. Two other essential methods are transform() and filter(), and understanding how they differ from agg() (and from each other) is key to writing idiomatic, efficient pandas code. transform() applies a function to each group but broadcasts the result back to the original DataFrame's shape — one output value per input row, not per group — making it ideal for adding group-relative columns like 'deviation from group mean' without a separate merge step. filter() instead evaluates a boolean condition on each group as a whole and either keeps or discards all rows of that group, which is useful for tasks like 'only keep customers with more than 3 orders'.
Cricket analogy: transform() is like giving every batter in an IPL team their strike rate relative to their team's average, one number per player, while filter() is like keeping or dropping an entire franchise based on whether it won more than 8 matches that season.
transform() for Row-Aligned Group Statistics
A classic transform use case is mean-centering or normalizing values within each group, e.g. subtracting the group mean from each row's value to see how far above or below average that row is. Because transform() returns an object with the same index and length as the input, its result can be assigned directly as a new column: df['dev_from_mean'] = df.groupby('region')['revenue'].transform('mean') followed by df['revenue'] - df['dev_from_mean'] gives per-row deviation, no join required. This is significantly cleaner than the older pattern of aggregating separately and merging back on the group key.
Cricket analogy: Subtracting each batter's runs from their team's average score, computed via groupby('team')['runs'].transform('mean'), reveals per-innings deviation from the team norm without a separate merge back to the scorecard.
import pandas as pd
df = pd.DataFrame({
'region': ['East', 'East', 'West', 'West', 'West'],
'revenue': [1200, 800, 950, 1600, 700]
})
# transform: same length as df, broadcasts group mean to every row
df['region_avg'] = df.groupby('region')['revenue'].transform('mean')
df['pct_of_region_avg'] = df['revenue'] / df['region_avg']
print(df)
# region revenue region_avg pct_of_region_avg
# 0 East 1200 1000.0 1.20
# 1 East 800 1000.0 0.80
# 2 West 950 1083.3 0.877
# 3 West 1600 1083.3 1.477
# 4 West 700 1083.3 0.646
# filter: keep only groups (regions) whose total revenue exceeds 2200
big_regions = df.groupby('region').filter(lambda g: g['revenue'].sum() > 2200)
print(big_regions['region'].unique()) # ['West']filter() for Whole-Group Selection
filter() takes a function that receives each group's sub-DataFrame and must return a single boolean; groups for which the function returns True are kept in their entirety (all original rows), and groups returning False are dropped completely. Unlike boolean masking on individual rows, filter() reasons about properties of the whole group — total size, sum, presence of a particular value — making it the natural tool for questions like 'keep only stores open for all 12 months' or 'drop customers with fewer than 3 transactions'.
Cricket analogy: filter() is like a selector who evaluates each squad's entire season record and keeps only the squads that played all their scheduled matches, discarding rosters with any cancelled fixtures.
A useful mental model: agg() answers 'what is true about each group?' (one row out), transform() answers 'what does this row look like relative to its group?' (same rows, enriched), and filter() answers 'should this entire group be kept?' (subset of original rows, unchanged values).
transform() functions must return either a scalar (broadcast to the group) or an array matching the group's length — returning a differently-sized array raises an error. This differs from apply(), which is more permissive but generally slower and less predictable in output shape.
- transform() returns a result aligned to the original DataFrame's shape, one value per row, unlike agg() which collapses each group.
- A common transform pattern is computing group-relative statistics (e.g. deviation from group mean) without a manual merge.
- filter() evaluates a group-level boolean condition and keeps or drops entire groups, preserving original row values.
- agg() reduces, transform() broadcasts, filter() selects — three distinct group operations with different output shapes.
- transform() functions must return a scalar or a same-length array per group; mismatched lengths raise errors.
- filter() is well suited to conditions like 'group size >= N' or 'group total exceeds threshold'.
Practice what you learned
1. What distinguishes groupby().transform() from groupby().agg()?
2. What does groupby().filter(func) do?
3. Which use case is best solved with transform() rather than agg()?
4. What must a function passed to groupby().transform() return for a group of length n?
5. Which statement best summarizes the three group operations agg, transform, and filter?
Was this page helpful?
You May Also Like
GroupBy Basics
Understand pandas' split-apply-combine model via groupby(), including grouping keys, selecting columns, and iterating over groups.
Aggregation Functions
Explore the built-in and custom aggregation functions available after groupby, including agg() for applying multiple statistics at once.
apply, map, and applymap
A practical comparison of pandas' three element-wise transformation tools — Series.map, DataFrame.apply, and the deprecated DataFrame.applymap — and when to reach for each.
Boolean Filtering
Learn how to filter DataFrame rows using boolean masks built from comparison and logical operators, including combining conditions and using isin and query for cleaner syntax.