Introduction
Hypothesis testing is a statistical framework for deciding whether an observed effect in data, such as a difference between two groups, is likely to reflect a real underlying pattern or could plausibly be explained by random chance, by starting from a null hypothesis that assumes no real effect and only rejecting that assumption when the evidence against it is strong enough.
Cricket analogy: Before crediting a new bowling grip with lower economy rates, a coach starts by assuming it makes no real difference, the null hypothesis, and only concludes the grip actually helps, the alternative hypothesis, if the observed improvement is too large to plausibly be random match-to-match variation.
Explanation
The p-value is the probability of seeing a result at least as extreme as the one observed if the null hypothesis (no real effect) were actually true; a small p-value means the observed result would be unlikely under that assumption, and if the p-value falls below a chosen significance level, commonly 0.05, the null hypothesis is rejected in favor of the alternative hypothesis.
Cricket analogy: A coach compares the observed improvement in economy rate against a threshold for how unlikely such a result would be if the grip truly made no difference, expressed as a p-value, and only rejects the assumption of no effect if that p-value falls below a chosen significance level such as 0.05.
Every hypothesis test carries two possible error risks: a Type I error, incorrectly rejecting a true null hypothesis (a false positive, concluding an effect exists when it does not), and a Type II error, incorrectly failing to reject a false null hypothesis (a false negative, missing a real effect); the chosen significance level directly controls the Type I error rate.
Cricket analogy: A coach can make two kinds of mistakes here: crediting a grip that actually does nothing (a false positive, or Type I error), or dismissing a grip that genuinely helps as if it were noise (a false negative, or Type II error), the same two risks every hypothesis test must balance.
Example
# Illustrative example: two-sample t-test on example data
from scipy import stats
group_a = [23, 25, 21, 24, 22] # example measurements
group_b = [27, 29, 26, 28, 25] # example measurements
t_stat, p_value = stats.ttest_ind(group_a, group_b)
print(t_stat, p_value)
# if p_value < 0.05, reject the null hypothesis of no differenceAnalysis
A p-value below the significance threshold does not mean the effect is large or practically important, only that it is unlikely to be pure chance given the null hypothesis; interpreting results correctly means checking the p-value against the chosen threshold, considering the effect size alongside it, and remembering that a p-value is not the probability that the null hypothesis itself is true.
Cricket analogy: A p-value below the threshold tells a coach the improvement is unlikely to be luck, not that the grip is a dramatic upgrade; the coach still has to look at the actual size of the economy-rate change alongside the p-value, the same care needed interpreting any hypothesis test result.
Key Takeaways
- The null hypothesis assumes no real effect; the alternative hypothesis is accepted only when evidence against the null is strong enough.
- The p-value is the probability of an observed (or more extreme) result occurring if the null hypothesis were true.
- A common significance level threshold is 0.05, below which the null hypothesis is rejected.
- A Type I error is a false positive (rejecting a true null); a Type II error is a false negative (missing a real effect).
- A significant p-value indicates an effect is unlikely due to chance, but does not by itself indicate the effect's practical size or importance.
Practice what you learned
1. What does the null hypothesis typically assume?
2. What does a p-value represent?
3. What is a Type I error?
4. What is a Type II error?
5. If a result is statistically significant (low p-value), what should also be considered?
Was this page helpful?
You May Also Like
Correlation vs Causation
Why two variables moving together does not mean one causes the other, how confounding variables create misleading correlations, and how causation is tested.
What Is EDA
What exploratory data analysis is, why analysts use summary statistics and visuals to understand a dataset before modeling, and what issues it uncovers.
Charts & When to Use Them
How to match a chart type, such as bar, line, scatter, or pie, to the specific comparison, trend, or relationship the underlying data is meant to show.