Pandas GroupBy Explained With Examples

1What Is Pandas GroupBy?

Pandas GroupBy is a method that splits a DataFrame into groups based on the values in one or more columns, applies a function to each group, and combines the results into a new Series or DataFrame. It answers questions like 'what is the average sale per region?' in a single expression.

The mental model is split-apply-combine: split the data into buckets by a key, apply an aggregation or transformation to each bucket independently, then combine the per-group answers back together. Once this pattern clicks, most summarisation tasks become one or two readable lines of code.

2Basic Syntax and a First Example

A GroupBy call has two parts: the grouping key and the aggregation. On its own, df.groupby('region') does almost nothing — it is lazy and simply records how rows should be partitioned. The work happens when you attach an aggregation.

• df.groupby('region')['sales'].sum() # total sales per region
• df.groupby('region')['sales'].mean() # average sale per region
• df.groupby('region').size() # rows per region
• df.groupby(['region', 'product'])['sales'].sum() # two-level grouping

3The Split-Apply-Combine Pattern

Every GroupBy follows the same three phases, and understanding them makes the API predictable rather than magical.

• Split: rows are partitioned into groups by the key, so all rows sharing a region land together.
• Apply: a function runs on each group independently — an aggregation collapses the group to one value, a transform reshapes it, a filter keeps or drops it.
• Combine: the per-group outputs are reassembled into a single Series or DataFrame indexed by the group keys.

4Running Multiple Aggregations With agg()

Real analysis rarely needs a single statistic. The .agg() method lets you compute several aggregations at once and control the output column names, which keeps downstream code clean.

• df.groupby('region')['sales'].agg(['sum', 'mean', 'count'])
• df.groupby('region').agg(total=('sales', 'sum'), avg=('sales', 'mean')) # named aggregation
• df.groupby('region').agg({'sales': 'sum', 'units': 'max'}) # per-column functions
• df.groupby('region')['sales'].agg(lambda s: s.max() - s.min()) # custom range function

5transform() vs aggregate()

Aggregation collapses each group to a single row, but sometimes you want a group statistic attached back to every original row. That is exactly what transform() does — it returns a result the same length as the input.

• df['region_avg'] = df.groupby('region')['sales'].transform('mean')
• df['pct_of_region'] = df['sales'] / df.groupby('region')['sales'].transform('sum')
• df['rank_in_region'] = df.groupby('region')['sales'].rank(ascending=False)

6Filtering Whole Groups

The filter() method keeps or discards entire groups based on a group-level condition. Unlike boolean indexing, which works row by row, filter evaluates a predicate once per group and returns all rows of the groups that pass.

• df.groupby('region').filter(lambda g: g['sales'].sum() > 10000)
• df.groupby('customer').filter(lambda g: len(g) >= 5) # keep repeat customers

7Common Mistakes to Avoid

A few recurring errors trip up newcomers and produce silently wrong results.

• Forgetting that GroupBy is lazy — printing df.groupby('x') shows an object, not data. Attach an aggregation.
• Leaving keys in the index when you need to merge — use as_index=False or reset_index().
• Ignoring NaN keys — rows with a missing group key are dropped by default. Pass dropna=False to keep them.
• Aggregating every column by accident — select the column of interest first, or you may aggregate text columns unexpectedly.
• Using a slow apply(lambda ...) when a built-in like .sum() or .mean() would run far faster on large data.

8Keeping GroupBy Fast

GroupBy is highly optimised when you use built-in aggregations, because they run in vectorised C code. Custom Python functions passed to apply run per group in the interpreter and can be dramatically slower on wide data.

• Prefer named string aggregations ('sum', 'mean') over apply(lambda ...) whenever possible.
• Convert repetitive string keys to the category dtype to speed up grouping and cut memory.
• Group on the smallest set of columns that answers your question.
• Use observed=True with categorical keys to avoid materialising unused category combinations.

9Key Takeaways

GroupBy becomes second nature once you internalise a few core ideas.

• GroupBy implements split-apply-combine: partition by key, apply a function, recombine.
• The call is lazy — data only materialises when you aggregate.
• Use agg() with named outputs for multiple, cleanly labelled statistics.
• transform() attaches group stats back to original rows; aggregate() collapses to one row per group.
• Watch for dropped NaN keys and keys stuck in the index.

10Frequently Asked Questions

Q: What is the difference between groupby and pivot_table in Pandas? A: Both summarise data, but groupby returns a Series or long-format DataFrame indexed by keys, while pivot_table reshapes results into a spreadsheet-style grid with keys spread across rows and columns. pivot_table is essentially a groupby plus an unstack, and it adds convenient options like margins for totals.

Q: Why does my GroupBy result show the keys as an index? A: By default Pandas moves the grouping keys into the index. Pass as_index=False to df.groupby(...), or call .reset_index() on the result, to keep them as regular columns for merging or plotting.

Q: How do I group by multiple columns? A: Pass a list of column names, such as df.groupby(['region', 'product']). This creates a hierarchical (MultiIndex) result with one level per grouping column, which you can flatten with reset_index().

Q: Does GroupBy include rows with missing keys? A: Not by default — rows whose grouping key is NaN are excluded. Pass dropna=False to df.groupby(...) to include a separate group for the missing values.