Aggregations and Statistics
NumPy's aggregation functions — sum, mean, std, var, min, max, argmin, argmax, median, percentile — reduce an array (or one axis of it) to a smaller result, and they run as compiled, vectorized operations that vastly outperform manual Python loops. Every one of them accepts an axis parameter to control which dimension is collapsed, and understanding this parameter is the key to using them correctly on multi-dimensional data such as tabular datasets shaped as (rows, columns).
Cricket analogy: np.mean and np.max on a season's scoring array reduce hundreds of ball-by-ball entries to one number, and specifying axis lets you collapse by innings versus by player when the data is shaped as (players, innings).
Variance, Standard Deviation, and ddof
np.var() and np.std() compute population statistics by default (dividing by N, the number of observations), which is correct when your data represents the entire population. For a sample intended to estimate a population parameter, you typically want the sample (unbiased) variant, which divides by N - 1 instead of N — set ddof=1 (delta degrees of freedom) to get this. Forgetting ddof=1 is a very common source of subtly wrong statistics when working with sampled data, especially since pandas defaults to ddof=1 while NumPy defaults to ddof=0, a frequent source of confusion when values don't match between the two libraries.
Cricket analogy: Computing np.std on a bowler's economy rates across a full season (population) divides by N, but estimating the standard deviation from a sample of just five matches to infer season-wide consistency needs ddof=1 for an unbiased estimate.
Handling NaN Values in Aggregations
Standard aggregation functions propagate NaN: if any element in the reduced axis is NaN, the result is NaN. NumPy provides NaN-aware counterparts — np.nansum, np.nanmean, np.nanstd, np.nanmedian — that ignore NaN values during the computation, which is essential when working with real-world data containing missing values. These functions are the NumPy-level building blocks that pandas' own missing-data-aware aggregations rely on internally.
Cricket analogy: If a rain-affected match leaves one innings' score as NaN, np.sum over the season propagates that NaN into the total, so np.nansum is needed to get a meaningful season total that ignores the missing match.
import numpy as np
data = np.array([[85, 90, np.nan], [70, 88, 95], [60, np.nan, 75]])
# Standard mean propagates NaN
print(np.mean(data, axis=0)) # [ 71.67 89. 85. ] -- wait, NaN propagates per column with NaN present
# actual: columns with a NaN produce NaN: [71.66666667 nan nan]
# NaN-aware mean ignores missing values
print(np.nanmean(data, axis=0)) # [71.66666667 89. 85. ]
# Population vs sample standard deviation
scores = np.array([88, 92, 79, 85, 91])
print(scores.std()) # ddof=0 (population), divides by N
print(scores.std(ddof=1)) # ddof=1 (sample), divides by N-1 -- matches pandas' default
# argmax/argmin return the INDEX of the extreme value
print(scores.argmax()) # 1 (index of 92)pandas' .std() and .var() default to ddof=1 (sample statistics), while NumPy's .std() and .var() default to ddof=0 (population statistics) -- this mismatch is one of the most common sources of 'why don't these numbers match' confusion when moving between the two libraries.
np.mean(data) on an array containing any NaN returns NaN for the whole reduction (or that slice, with axis specified) -- always reach for np.nanmean/np.nansum/np.nanstd when your data may contain missing values, or filter/impute first.
- Aggregation functions (sum, mean, std, var, min, max) accept an axis parameter to control which dimension collapses.
- np.std/np.var default to ddof=0 (population); pass ddof=1 for the sample (unbiased) estimator.
- pandas defaults to ddof=1, unlike NumPy's ddof=0 default, a common source of mismatched results.
- Standard aggregations propagate NaN; use nan-prefixed variants (nanmean, nansum, etc.) to ignore missing values.
- argmin/argmax return the index of the extreme value, not the value itself.
- np.percentile and np.median provide robust central-tendency measures less sensitive to outliers than the mean.
Practice what you learned
1. What is the default value of ddof used by NumPy's std() and var() functions?
2. Why might np.std(data) and a pandas Series' .std() give different results on the same data?
3. What does np.mean() return for an axis/array containing a NaN value, using the standard (non-nan-prefixed) function?
4. What does scores.argmax() return?
5. Which function should you use to compute a mean while ignoring missing values encoded as NaN?
Was this page helpful?
You May Also Like
Vectorized Operations and Broadcasting
Vectorization applies operations across whole arrays without explicit loops, and broadcasting extends this to arrays of different but compatible shapes.
Handling Missing Data in Pandas
Learn how pandas represents missing values with NaN and NA, and the core toolkit — isna, dropna, fillna, interpolate — for detecting and handling them.
Aggregation Functions
Explore the built-in and custom aggregation functions available after groupby, including agg() for applying multiple statistics at once.
Linear Algebra with NumPy
Learn how NumPy represents vectors and matrices and how to perform matrix multiplication, transposition, inversion, determinants, and eigen-decomposition using the linalg module.