Introduction
A/B testing compares two versions of something, typically labeled A (the control, or existing version) and B (the treatment, or new version), by randomly assigning different users to each version and measuring a chosen outcome, such as click-through rate or conversion rate. The random assignment is the crucial part: because users are split into groups by chance rather than by any characteristic of their own, any consistent difference in outcome between the groups can be attributed to the change being tested rather than to pre-existing differences between the kinds of users who ended up in each group.
Cricket analogy: A coach testing a new batting stance doesn't let players choose which stance to try; players are randomly assigned to the old or new stance, so any difference in scoring afterward reflects the stance itself rather than which players happened to prefer trying something new, just as A/B testing relies on random assignment.
Explanation
After running the test and observing a difference between group A's and group B's conversion rates, the next question is whether that difference reflects a real effect of the change or could plausibly have arisen just from random variation in which users happened to land in each group. A statistical significance test, most commonly a two-proportion z-test or chi-squared test for conversion-rate comparisons, computes a p-value: the probability of seeing a difference at least as large as the one observed if there were truly no difference between A and B. A small p-value, conventionally below 0.05, is taken as evidence that the observed difference is unlikely to be due to chance alone.
Cricket analogy: Seeing one net-session's batting average beat another's by a few runs doesn't prove a new stance works; a significance test asks how likely that gap is from pure day-to-day variation if the stance made no real difference, the same logic behind an A/B test's p-value.
A p-value below the chosen threshold does not, by itself, guarantee the effect is large enough to matter practically, since with a large enough sample size even a tiny, commercially irrelevant difference can become statistically significant; this is why effect size, the actual magnitude of the difference such as a two-percentage-point lift in conversion, should always be reported alongside the p-value. Sample size must also be planned before running the test through a power analysis, which estimates how many users are needed in each group to reliably detect an effect of a meaningful size, since a test that is stopped too early or run with too few users risks concluding there is no effect when a real one simply could not be detected with the available data.
Cricket analogy: A statistically significant half-run improvement in average across a huge number of net sessions might be real but too small to matter for team selection; effect size alongside significance is what matters, and a coach needs enough net sessions planned upfront to reliably detect a meaningful change, the same logic behind power analysis.
Example
import numpy as np
from statsmodels.stats.proportion import proportions_ztest
# Illustrative example counts, not real data
conversions = np.array([120, 150]) # [A, B]
sample_sizes = np.array([2000, 2000]) # [A, B]
z_stat, p_value = proportions_ztest(conversions, sample_sizes)
lift = conversions[1] / sample_sizes[1] - conversions[0] / sample_sizes[0]
print(f"lift: {lift:.3%}, p-value: {p_value:.4f}")Analysis
In this illustrative example, group A converts at 6.0 percent and group B at 7.5 percent, a lift of 1.5 percentage points; whether that lift is reported as statistically significant depends entirely on the p_value computed by proportions_ztest, which accounts for both the size of the lift and the sample size in each group. If the same 1.5-point lift had been observed with sample sizes of 200 rather than 2000 per group, the same test would very likely produce a much larger p-value, illustrating that a promising-looking raw difference is not evidence of a real effect until it has been checked against a properly sized, randomly assigned test.
Cricket analogy: In an illustrative example, a coach's small net-group trial might show one stance scoring higher, but with only a handful of sessions the same test would likely show a much weaker significance than the identical lift measured across a full season's worth of sessions.
Key Takeaways
- A/B testing randomly assigns users to a control (A) and a treatment (B) version to isolate the effect of the change being tested.
- Random assignment is essential so any consistent outcome difference can be attributed to the change rather than pre-existing group differences.
- A p-value estimates the probability of seeing an observed difference or larger if there were truly no effect; a small p-value suggests the difference is unlikely due to chance.
- Statistical significance does not guarantee practical importance; effect size should always be reported alongside the p-value.
- Sample size should be planned in advance through a power analysis so the test can reliably detect a meaningful effect.
Practice what you learned
1. Why is random assignment essential in an A/B test?
2. What does a p-value represent in an A/B test?
3. Why can a statistically significant result still be practically unimportant?
4. What is the purpose of a power analysis before running an A/B test?
Was this page helpful?
You May Also Like
Outlier Detection
How the IQR rule and z-score method identify data points that fall unusually far from the rest of a dataset.
Sampling Methods
How random, stratified, and systematic sampling techniques select a representative subset of a larger population for analysis.
DataFrames Explained
How a DataFrame organizes labeled rows and columns of data in memory and enables filtering, grouping, and joining operations.