Hypothesis Testing Cheat Sheet
Core statistical hypothesis testing concepts and workflows, covering t-tests, chi-square tests, p-values, and confidence intervals with scipy.
2 PagesBeginnerMar 8, 2026
T-Tests
One-sample, independent, and paired t-tests.
python
from scipy import statsgroup_a = [23, 25, 21, 22, 24, 20, 26]group_b = [28, 30, 27, 29, 31, 26, 32]# One-sample t-test: is the mean different from a known value?t_stat, p_val = stats.ttest_1samp(group_a, popmean=25)# Independent two-sample t-test (Welch's, unequal variance assumed)t_stat, p_val = stats.ttest_ind(group_a, group_b, equal_var=False)# Paired t-test: same subjects measured twice (before/after)before = [10, 12, 9, 11, 14]after = [12, 13, 10, 13, 15]t_stat, p_val = stats.ttest_rel(before, after)print(f"t = {t_stat:.3f}, p = {p_val:.4f}")
Chi-Square, ANOVA & Non-Parametric Tests
Tests for categorical and non-normal data.
python
from scipy.stats import chi2_contingency, f_oneway, mannwhitneyu# Chi-square test of independence (categorical vs categorical)table = [[50, 30], [20, 40]] # contingency tablechi2, p, dof, expected = chi2_contingency(table)# One-way ANOVA: compare means across 3+ groupsgroup_c = [22, 24, 23, 25]f_stat, p_val = f_oneway([23, 25, 21, 22], [28, 30, 27, 29], group_c)# Mann-Whitney U: non-parametric alternative to the t-test (no normality assumption)u_stat, p_val = mannwhitneyu([23, 25, 21, 22], [28, 30, 27, 29])
Core Terms
Vocabulary used across hypothesis tests.
- Null hypothesis (H0)- the default assumption of 'no effect' or 'no difference'
- Alternative hypothesis (H1)- what you're trying to find evidence for
- p-value- probability of observing data this extreme (or more) if H0 is true
- Significance level (α)- threshold for rejecting H0, typically 0.05
- Type I error- rejecting a true H0 (false positive), rate = α
- Type II error- failing to reject a false H0 (false negative), rate = β
- Confidence interval- range of plausible values for a parameter at a given confidence level (e.g. 95%)
- One-tailed vs two-tailed- whether the test checks for an effect in one direction only or either direction
Choosing the Right Test
Match the test to your data type and design.
- Compare two group means, normal data- independent samples t-test
- Compare two group means, before/after- paired t-test
- Compare 3+ group means- one-way ANOVA (then post-hoc tests like Tukey HSD)
- Compare two proportions/categorical counts- chi-square test of independence
- Compare two groups, non-normal/ordinal data- Mann-Whitney U test
- Test correlation between two continuous variables- Pearson (linear) or Spearman (monotonic) correlation test
Statistical Power & Sample Size
Compute required sample size or achieved power before running a test.
python
from statsmodels.stats.power import TTestIndPowerfrom statsmodels.stats.proportion import proportion_effectsize, samplesize_proportions_2indep_onetailanalysis = TTestIndPower()# Required n per group to detect Cohen's d=0.5 at alpha=0.05, power=0.8n_needed = analysis.solve_power(effect_size=0.5, alpha=0.05, power=0.8, ratio=1.0)print(f"n per group: {n_needed:.0f}")# Achieved power given a fixed sample sizeachieved_power = analysis.solve_power(effect_size=0.5, nobs1=40, alpha=0.05, ratio=1.0)print(f"power: {achieved_power:.3f}")# Sample size for a two-proportion test (e.g. conversion rate A/B)effect = proportion_effectsize(0.10, 0.12) # baseline 10% -> 12%n_prop = samplesize_proportions_2indep_onetail(effect, prop1=0.10, alpha=0.05, power=0.8)print(f"n per arm: {n_prop:.0f}")
Multiple Comparisons Correction
Control false positives when running many hypothesis tests at once.
python
from statsmodels.stats.multitest import multipletestsp_values = [0.001, 0.02, 0.03, 0.04, 0.049, 0.08, 0.12, 0.30]# Bonferroni: strict, controls family-wise error rate (FWER)reject, p_bonf, _, _ = multipletests(p_values, alpha=0.05, method="bonferroni")# Holm: uniformly more powerful than Bonferroni, still controls FWERreject_holm, p_holm, _, _ = multipletests(p_values, alpha=0.05, method="holm")# Benjamini-Hochberg: controls false discovery rate (FDR), less conservative,# preferred when running many tests (e.g. genomics, feature screening)reject_fdr, p_fdr, _, _ = multipletests(p_values, alpha=0.05, method="fdr_bh")print("BH-adjusted:", p_fdr.round(3))print("Rejected H0:", reject_fdr)
Permutation Test
Distribution-free test that makes no normality assumption.
python
import numpy as npfrom scipy import statsrng = np.random.default_rng(42)group_a = np.array([23, 25, 21, 22, 24, 20, 26])group_b = np.array([28, 30, 27, 29, 31, 26, 32])def statistic(x, y): return np.mean(x) - np.mean(y)# scipy's built-in permutation test (exact or Monte Carlo)res = stats.permutation_test( (group_a, group_b), statistic, n_resamples=10000, alternative="two-sided", random_state=rng)print(f"observed diff = {res.statistic:.3f}, p = {res.pvalue:.4f}")# Manual version: shuffle labels, recompute the statistic, compare to observedpooled = np.concatenate([group_a, group_b])n_a = len(group_a)observed = statistic(group_a, group_b)perm_diffs = np.empty(10000)for i in range(10000): shuffled = rng.permutation(pooled) perm_diffs[i] = statistic(shuffled[:n_a], shuffled[n_a:])p_manual = np.mean(np.abs(perm_diffs) >= np.abs(observed))
Bootstrap Confidence Intervals
Estimate a CI for any statistic without assuming a sampling distribution.
python
import numpy as npfrom scipy import statsdata = np.array([23, 25, 21, 22, 24, 20, 26, 27, 19, 24])# scipy.stats.bootstrap: BCa (bias-corrected accelerated) interval by defaultres = stats.bootstrap( (data,), np.median, confidence_level=0.95, n_resamples=9999, method="BCa")print(f"95% CI for median: ({res.confidence_interval.low:.2f}, {res.confidence_interval.high:.2f})")# Manual percentile bootstrap for a custom statistic (e.g. trimmed mean)rng = np.random.default_rng(0)boot_stats = np.array([ stats.trim_mean(rng.choice(data, size=len(data), replace=True), 0.1) for _ in range(9999)])lo, hi = np.percentile(boot_stats, [2.5, 97.5])
Effect Size Reference
Statistical significance ≠ practical significance — always report an effect size alongside p-values.
- Cohen's d- standardized mean difference; (mean1-mean2)/pooled_std; 0.2=small, 0.5=medium, 0.8=large
- Hedges' g- bias-corrected variant of Cohen's d, preferred for small samples (n < 20 per group)
- Eta squared (η²)- proportion of variance explained by group membership in ANOVA; 0.01=small, 0.06=medium, 0.14=large
- Cramer's V- effect size for chi-square tests of association between categorical variables, range [0, 1]
- Odds ratio- ratio of odds of an outcome between two groups; used for 2x2 tables and logistic models
- r (rank-biserial)- effect size companion to Mann-Whitney U, comparable across non-parametric tests
- Number needed to treat (NNT)- 1/absolute risk reduction; how many subjects must be treated to prevent one additional bad outcome
Pro Tip
A p-value tells you the probability of the data given the null hypothesis, not the probability the null hypothesis is true — don't say 'there's a 95% chance treatment works,' say 'if there were no effect, data this extreme would occur less than 5% of the time.'
Was this cheat sheet helpful?
Explore Topics
#HypothesisTesting#HypothesisTestingCheatSheet#DataScience#Beginner#TTests#Chi#Square#ANOVA#MachineLearning#Testing#CheatSheet#SkillVeris
