What Is a p-value Explained Simply
SkillVeris Team
Data Science Team
A p-value is the probability of seeing results at least as extreme as yours if the null hypothesis (no real effect) were true.
In this guide, you'll learn:
- A small p-value means the data would be surprising under the assumption of no effect, so you doubt that assumption.
- The common 0.05 threshold is a convention, not a law of nature, and choosing it before you look at the data matters.
- A p-value is not the probability that your hypothesis is true, nor the size of an effect.
- Statistical significance and practical importance are different things a tiny effect can be significant with enough data.
1What Is a p-value?
A p-value is the probability of observing results at least as extreme as the ones you got, assuming the null hypothesis is true. The null hypothesis usually says there is no real effect, no difference, or no relationship just random noise. A small p-value tells you your data would be unusual in that no-effect world, which gives you reason to doubt it.
Think of it as a surprise meter. If you flip a coin ten times and get ten heads, that outcome is very surprising for a fair coin, so the p-value would be tiny. It does not prove the coin is rigged, but it makes fairness hard to believe.
2The Intuition Behind It
P-values come from hypothesis testing, a framework for deciding whether an observed pattern is likely real or just chance. You start by assuming nothing is going on, then ask how weird your actual data looks under that assumption.
The logic is a form of proof by contradiction. If the data would be extremely rare under the null hypothesis, you reject the null and conclude something real is probably happening. If the data is unremarkable, you fail to reject it you simply do not have enough evidence to claim an effect.
💡Quick Mental Model
Read a p-value as: 'If nothing interesting were happening, how often would I see data this extreme by luck alone?' The rarer, the more suspicious of the null.
3How to Read a p-value
P-values range from 0 to 1. Lower values mean stronger evidence against the null hypothesis. Researchers often compare the p-value to a pre-chosen threshold called alpha, commonly 0.05.
- p < 0.01 strong evidence against the null hypothesis; the result is very unlikely under chance alone.
- p < 0.05 conventionally 'statistically significant'; often used as a decision cutoff.
- p between 0.05 and 0.10 weak or borderline evidence; treat with caution.
- p > 0.10 little evidence against the null; the data is broadly consistent with no effect.
⚠️Not a Magic Line
The 0.05 cutoff is a convention popularized decades ago, not a fundamental truth. A p-value of 0.049 is not meaningfully different from 0.051.
4A Worked Example
Suppose you run an A/B test on a checkout button. Version A converts at 10 percent, version B at 12 percent. Is B genuinely better, or did the difference come from random variation among your visitors?
Running the Test
You run a two-proportion test comparing the conversion rates. The null hypothesis says the true conversion rates are equal. The test returns a p-value based on your sample sizes and the observed gap.
from scipy.stats import chi2_contingency
table = [[100, 900], [120, 880]] # conversions vs non-conversions
chi2, p, dof, expected = chi2_contingency(table)
print(p) # e.g. 0.03 -> reject the null at alpha = 0.05Interpreting the Result
A p-value of 0.03 means that if the two versions truly converted at the same rate, you would see a gap this large or larger only about 3 percent of the time. That is unusual enough to conclude B likely performs better though you should still check the effect size and confidence interval before rolling it out.
5What a p-value Is Not
Most misunderstandings come from reading a p-value as something it is not. It answers a narrow question, and stretching it beyond that leads to bad conclusions.
- It is NOT the probability that the null hypothesis is true.
- It is NOT the probability that your results happened by chance.
- It is NOT a measure of how big or important an effect is.
- It is NOT proof of anything a low p-value is evidence, not certainty.
6Significance Versus Importance
Statistical significance and practical importance are two separate ideas that people constantly conflate. A result can be statistically significant yet far too small to matter in the real world.
With a very large sample, even a trivial difference such as a 0.1 percent change in conversion can produce a tiny p-value. That is why you should always report the effect size and a confidence interval alongside the p-value. The p-value tells you whether an effect probably exists; the effect size tells you whether you should care.
🔑Report Both
Always pair a p-value with an effect size and confidence interval. Significance answers 'is it real?'; effect size answers 'does it matter?'
7Common Mistakes to Avoid
P-values are easy to misuse, and several recurring errors quietly invalidate results. Watch for these.
- P-hacking: trying many analyses and reporting only the ones that cross 0.05. This manufactures false positives.
- Ignoring multiple comparisons: run 20 tests and, by chance, roughly one will hit p < 0.05 even with no real effect. Apply corrections like Bonferroni.
- Choosing the threshold after seeing the data pick alpha before you look, not after.
- Treating p > 0.05 as proof of no effect absence of evidence is not evidence of absence.
- Reporting significance without effect size, so readers cannot judge whether the finding matters.
8Key Takeaways
The core ideas about p-values fit into a handful of durable points.
- A p-value is the chance of data this extreme if the null hypothesis were true.
- Small p-value means the data is surprising under 'no effect', so you doubt that assumption.
- 0.05 is a convention decide your threshold before you look at the data.
- A p-value is not the probability your hypothesis is true, nor a measure of effect size.
- Always report effect size and confidence intervals alongside the p-value.
9Frequently Asked Questions
Q: What does a p-value of 0.05 actually mean? A: It means that if the null hypothesis were true, you would observe data at least this extreme about 5 percent of the time by chance. It is a threshold many fields use to call a result statistically significant, but it is a convention rather than a hard rule.
Q: Does a small p-value prove my hypothesis is correct? A: No. A small p-value is evidence against the null hypothesis, not proof that your alternative explanation is true. Other factors like sample bias, confounding variables, or multiple testing can also produce small p-values.
Q: Can a result be statistically significant but unimportant? A: Yes. With a large enough sample, even a tiny, practically meaningless difference can produce a small p-value. Always check the effect size to judge whether the result matters in practice.
Q: What is p-hacking? A: P-hacking is running many analyses or tests and reporting only those that reach significance. It inflates false positives and is a major cause of results that fail to replicate. Pre-registering your analysis plan helps prevent it.
Related Reading
Get The Print Version
Download a PDF of this article for offline reading.
About the Publisher
SkillVeris Team
Data Science Team
Our data team shares real-world analytics, ML, and SQL insights grounded in industry practice.
View all postsRelated Posts
Never miss an update
Get the latest tutorials and guides delivered to your inbox.
No spam. Unsubscribe anytime.