Hypothesis Testing Cheat Sheet

Core statistical hypothesis testing concepts and workflows, covering t-tests, chi-square tests, p-values, and confidence intervals with scipy.

2 PagesBeginnerMar 8, 2026

T-Tests

One-sample, independent, and paired t-tests.

python

from scipy import statsgroup_a = [23, 25, 21, 22, 24, 20, 26]group_b = [28, 30, 27, 29, 31, 26, 32]# One-sample t-test: is the mean different from a known value?t_stat, p_val = stats.ttest_1samp(group_a, popmean=25)# Independent two-sample t-test (Welch's, unequal variance assumed)t_stat, p_val = stats.ttest_ind(group_a, group_b, equal_var=False)# Paired t-test: same subjects measured twice (before/after)before = [10, 12, 9, 11, 14]after = [12, 13, 10, 13, 15]t_stat, p_val = stats.ttest_rel(before, after)print(f"t = {t_stat:.3f}, p = {p_val:.4f}")

Chi-Square, ANOVA & Non-Parametric Tests

Tests for categorical and non-normal data.

python

from scipy.stats import chi2_contingency, f_oneway, mannwhitneyu# Chi-square test of independence (categorical vs categorical)table = [[50, 30], [20, 40]]  # contingency tablechi2, p, dof, expected = chi2_contingency(table)# One-way ANOVA: compare means across 3+ groupsgroup_c = [22, 24, 23, 25]f_stat, p_val = f_oneway([23, 25, 21, 22], [28, 30, 27, 29], group_c)# Mann-Whitney U: non-parametric alternative to the t-test (no normality assumption)u_stat, p_val = mannwhitneyu([23, 25, 21, 22], [28, 30, 27, 29])

Core Terms

Vocabulary used across hypothesis tests.

Null hypothesis (H0)- the default assumption of 'no effect' or 'no difference'
Alternative hypothesis (H1)- what you're trying to find evidence for
p-value- probability of observing data this extreme (or more) if H0 is true
Significance level (α)- threshold for rejecting H0, typically 0.05
Type I error- rejecting a true H0 (false positive), rate = α
Type II error- failing to reject a false H0 (false negative), rate = β
Confidence interval- range of plausible values for a parameter at a given confidence level (e.g. 95%)
One-tailed vs two-tailed- whether the test checks for an effect in one direction only or either direction

Choosing the Right Test

Match the test to your data type and design.

Compare two group means, normal data- independent samples t-test
Compare two group means, before/after- paired t-test
Compare 3+ group means- one-way ANOVA (then post-hoc tests like Tukey HSD)
Compare two proportions/categorical counts- chi-square test of independence
Compare two groups, non-normal/ordinal data- Mann-Whitney U test
Test correlation between two continuous variables- Pearson (linear) or Spearman (monotonic) correlation test

Pro Tip

A p-value tells you the probability of the data given the null hypothesis, not the probability the null hypothesis is true — don't say 'there's a 95% chance treatment works,' say 'if there were no effect, data this extreme would occur less than 5% of the time.'

Was this cheat sheet helpful?

A/B Testing Cheat Sheet

Probability Distributions Cheat Sheet