Introduction
An outlier is a data point that differs substantially from most of the other observations in a dataset, and identifying outliers matters because they can either represent genuine rare events worth investigating or data-entry errors that would distort an analysis if left unchecked. Outlier detection is not about automatically deleting unusual values; it is about flagging them so an analyst can decide, with domain knowledge, whether a value is a mistake to correct, a rare-but-real event to keep, or a signal that the dataset needs further investigation before conclusions are drawn from it.
Cricket analogy: A score of 400 not out in an innings isn't automatically struck from the record book; a selector investigates whether it's a genuine feat worth noting or a scoring error, the same judgment call an analyst makes before treating a statistical outlier as an error or a real event.
Explanation
The interquartile range, or IQR, method defines an outlier using the spread of the middle half of the data. First, the first quartile Q1 (the value below which 25 percent of the data falls) and the third quartile Q3 (below which 75 percent falls) are computed, and IQR is defined as Q3 minus Q1. A common rule then flags any value below Q1 minus 1.5 times IQR, or above Q3 plus 1.5 times IQR, as an outlier. This method is popular because it does not assume the data follows a particular distribution, and it is resistant to the influence of the very outliers it is trying to detect, since Q1 and Q3 are based on the middle of the data rather than its extremes.
Cricket analogy: Ranking a squad's batting averages and marking the middle fifty percent's spread as normal, then flagging anyone far below or above that band as unusual, mirrors the IQR rule, which uses the spread of the middle half of a dataset rather than its extremes to define what counts as an outlier.
The z-score method instead measures how many standard deviations a value sits from the dataset's mean, computed as z equals the value minus the mean, divided by the standard deviation; a common convention flags values with an absolute z-score above 3 as outliers. Unlike the IQR method, z-scores assume the data is roughly normally distributed, and because both the mean and standard deviation used in the formula are themselves calculated from the full dataset including any outliers, extreme values can inflate the standard deviation and mask their own extremity, a known weakness compared to the IQR approach.
Cricket analogy: Measuring how many 'typical spreads' a batting average sits from the squad's mean works well if scores cluster normally, but one freakishly high score inflates the spread used in the calculation itself, masking its own extremity, a known weakness of the z-score method.
Example
import pandas as pd
data = pd.Series([12, 14, 13, 15, 14, 13, 95, 12, 14])
# IQR method
q1, q3 = data.quantile(0.25), data.quantile(0.75)
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
iqr_outliers = data[(data < lower) | (data > upper)]
# Z-score method
z_scores = (data - data.mean()) / data.std()
z_outliers = data[z_scores.abs() > 3]Analysis
In this illustrative example series, the value 95 sits far above the other values, which cluster tightly around 12 to 15; the IQR method flags it clearly because Q1 and Q3 stay anchored to the tight cluster regardless of the single extreme value. The z-score calculation, however, uses a mean and standard deviation that are themselves pulled upward and widened by the value 95, which can reduce its computed z-score compared to what it would be if 95 were excluded from the calculation, illustrating why the IQR method is often preferred as a first check on small or heavily skewed datasets, with z-scores reserved for data already known to be roughly normally distributed.
Cricket analogy: In an illustrative example set of averages clustered tightly with one huge outlying score, the IQR rule flags it cleanly because Q1 and Q3 stay anchored to the tight cluster, while the outlier itself pulls the mean and spread used in a z-score calculation, weakening its own detection.
Key Takeaways
- An outlier is a value substantially different from the rest of the data; detection flags it for review, it does not automatically delete it.
- The IQR method flags values below Q1 minus 1.5×IQR or above Q3 plus 1.5×IQR, using the middle-half spread of the data.
- The IQR method does not assume a particular distribution and resists distortion by the outliers it detects.
- The z-score method flags values with an absolute z-score above roughly 3, based on distance from the mean in standard deviations.
- Z-scores assume roughly normal data and can be distorted because extreme values inflate the mean and standard deviation used in their own calculation.
Practice what you learned
1. In the IQR method, what defines the lower bound below which a value is flagged as an outlier?
2. Why is the IQR method described as resistant to the outliers it detects?
3. What assumption does the z-score method make that the IQR method does not?
4. Why can extreme values reduce their own computed z-score?
Was this page helpful?
You May Also Like
DataFrames Explained
How a DataFrame organizes labeled rows and columns of data in memory and enables filtering, grouping, and joining operations.
Sampling Methods
How random, stratified, and systematic sampling techniques select a representative subset of a larger population for analysis.
A/B Testing Basics
How splitting users into control and treatment groups and comparing outcomes with statistical significance testing reveals a change's true effect.