What is A/B testing and how do you design a valid experiment?
Learn what A/B testing is and how to design a valid experiment: hypothesis, randomization, power analysis, and significance testing, with Python examples.
Expected Interview Answer
A/B testing is a controlled experiment that randomly splits users into a control group (A) and a treatment group (B) to measure whether a change causes a statistically significant difference in a chosen metric.
A valid design starts with a clear hypothesis and one primary success metric, then uses a power analysis to fix the sample size and run length before launch. Users are randomized so the groups are comparable, the change is isolated to a single variable, and results are evaluated with a significance test only after the pre-committed sample is reached, guarding against peeking, novelty effects, and multiple-comparison inflation.
- Establishes causation, not just correlation
- Quantifies impact with confidence intervals
- Reduces risk before a full rollout
- Removes guesswork and opinion from decisions
- Makes results reproducible and auditable
AI Mentor Explanation
A/B testing is like giving two evenly matched batting lineups the same pitch and bowling attack, changing only the bat coating for one side, then comparing average runs over many innings. Randomizing which players go to each group keeps talent balanced, so any run difference can be credited to the bat and not to who happened to face the tougher spell.
Step-by-Step Explanation
Step 1
State a hypothesis
Define the change and the expected direction, e.g. the new layout increases signups.
Step 2
Choose one primary metric
Pick a single success metric and guardrail metrics to catch harm.
Step 3
Run a power analysis
Compute the sample size and duration for the minimum detectable effect at your alpha and power.
Step 4
Randomize assignment
Split users randomly and check the groups are balanced on key covariates.
Step 5
Isolate one variable
Change only the treatment so any effect can be attributed to it.
Step 6
Analyze at the pre-set sample
Run the significance test only after the planned sample, then report effect size and confidence interval.
What Interviewer Expects
- A clear hypothesis and single primary metric
- Understanding of randomization and comparable groups
- Power analysis and pre-committed sample size
- Significance testing with p-values and confidence intervals
- Awareness of peeking, novelty effects and multiple comparisons
Common Mistakes
- Stopping the test early the moment it looks significant (peeking)
- Not computing sample size before launch
- Changing several variables at once so effects cannot be separated
- Ignoring guardrail metrics and only watching the win metric
- Treating a non-significant result as proof of no effect
Best Answer (HR Friendly)
“A/B testing shows two versions of something to two random groups of users and measures which performs better on a goal like signups or sales. Because the groups are chosen at random and only one thing changes, any difference in results can be trusted to come from that change.”
Code Example
from statsmodels.stats.proportion import proportions_ztest, proportion_effectsize
from statsmodels.stats.power import NormalIndPower
# Observed conversions
conversions = [1200, 1310] # control A, treatment B
visitors = [12000, 12000]
stat, pvalue = proportions_ztest(conversions, visitors)
print('p-value:', round(pvalue, 4))
# Sample size for a 1% absolute lift, 80% power, alpha 0.05
effect = proportion_effectsize(0.10, 0.11)
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8)
print('needed per group:', int(n))Follow-up Questions
- Why is peeking at results a problem, and how do sequential tests help?
- How do you pick the minimum detectable effect for a power analysis?
- What is a guardrail metric and why do you track one?
- How does the novelty effect distort early A/B results?
- When would you use a multi-armed bandit instead of a fixed A/B test?
MCQ Practice
1. Why are users randomly assigned to groups?
Randomization balances confounders so differences can be attributed to the treatment.
2. What does a power analysis determine before launch?
Power analysis fixes the sample size needed to detect a given effect at a chosen alpha and power.
3. Which practice invalidates an A/B test?
Peeking and stopping early inflates the false-positive rate.
Flash Cards
What is A/B testing? — A randomized controlled experiment comparing control (A) and treatment (B) on one primary metric.
Why randomize? — It balances confounders so any metric difference can be attributed to the change, establishing causation.
Power analysis — Pre-launch calculation of sample size and duration for a minimum detectable effect at set alpha and power.
Peeking problem — Repeatedly checking and stopping when significant inflates false positives; analyze at the pre-set sample.
Guardrail metric — A secondary metric watched to ensure the winning change does not harm the overall experience.