A/B Testing Cheat Sheet
A practical guide to designing, running, and statistically analyzing A/B tests, covering sample size, significance testing, and common pitfalls.
2 PagesIntermediateMar 10, 2026
Sample Size & Z-Test
Power analysis and two-proportion z-test.
python
import numpy as npfrom statsmodels.stats.power import NormalIndPowerfrom statsmodels.stats.proportion import proportions_ztest# Sample size needed to detect an effect (power analysis)analysis = NormalIndPower()n_per_group = analysis.solve_power( effect_size=0.2, # standardized effect size (Cohen's h) alpha=0.05, # significance level power=0.8, # desired statistical power ratio=1.0,)print(f"Required sample size per group: {n_per_group:.0f}")# Two-proportion z-test for conversion rate comparisonconversions = np.array([120, 150]) # [control, treatment]visitors = np.array([2000, 2000])z_stat, p_value = proportions_ztest(conversions, visitors)print(f"z = {z_stat:.3f}, p = {p_value:.4f}")
T-Test for Continuous Metrics
Compare means and build a confidence interval.
python
from scipy import statsimport numpy as np# For continuous metrics (e.g. revenue per user), use a t-testcontrol = np.random.normal(50, 10, 1000)treatment = np.random.normal(52, 10, 1000)t_stat, p_value = stats.ttest_ind(control, treatment, equal_var=False)print(f"t = {t_stat:.3f}, p = {p_value:.4f}")# 95% confidence interval for the difference in meansdiff = treatment.mean() - control.mean()se = np.sqrt(control.var(ddof=1) / len(control) + treatment.var(ddof=1) / len(treatment))ci = (diff - 1.96 * se, diff + 1.96 * se)print(f"Difference: {diff:.2f}, 95% CI: {ci}")
A/B Testing Concepts
Key vocabulary for designing experiments.
- Null hypothesis (H0)- assumes no difference between control and treatment
- Statistical significance (p-value)- probability of seeing this result (or more extreme) if H0 were true
- Statistical power- probability of correctly detecting a true effect (typically target 80%)
- Minimum Detectable Effect (MDE)- smallest effect size the test is designed to reliably detect
- Sample Ratio Mismatch (SRM)- unequal group sizes vs. the intended split, signals a bug in randomization
- Novelty effect- short-term behavior change from users noticing something new, not a lasting effect
- Peeking problem- repeatedly checking significance before the planned sample size inflates false positive rate
- Multiple comparisons- testing many metrics/segments increases chance of a false positive; use a correction (e.g. Bonferroni)
Test Design Checklist
Steps to run a trustworthy experiment.
- Define one primary metric- pick a single primary success metric before launch to avoid p-hacking
- Compute required sample size- run a power analysis before starting, don't stop early
- Randomize consistently- assign users to variants deterministically (e.g. hash of user ID)
- Check for SRM- verify observed group sizes match the intended split before trusting results
- Run for full business cycles- avoid day-of-week effects by running at least 1-2 full weeks
- Pre-register the analysis plan- decide the test and segments in advance to avoid fishing for significance
Pro Tip
Decide your sample size and test duration up front using a power analysis, and don't stop the test early just because you see a significant p-value — 'peeking' at results repeatedly inflates your false positive rate far above 5%.
Was this cheat sheet helpful?
Explore Topics
#ABTesting#ABTestingCheatSheet#DataScience#Intermediate#SampleSizeZTest#TTestForContinuousMetrics#ABTestingConcepts#TestDesignChecklist#MachineLearning#Testing#CheatSheet#SkillVeris
