What is a p-value and how do you interpret it?
Understand what a p-value means, how to interpret it against alpha, and the common mistakes to avoid in hypothesis testing and A/B testing.
Expected Interview Answer
A p-value is the probability of observing results at least as extreme as the ones you got, assuming the null hypothesis is true. A small p-value suggests your data would be unlikely under the null hypothesis, giving evidence against it.
You compare the p-value to a pre-chosen significance level, alpha (commonly 0.05). If the p-value is less than or equal to alpha, you reject the null hypothesis and call the result statistically significant; otherwise you fail to reject it. Crucially, the p-value is not the probability that the null hypothesis is true, nor the probability that your finding happened by chance, and it says nothing about the size or practical importance of an effect. A tiny p-value with a trivial effect size can still be practically meaningless.
- Provides an objective threshold for decision making
- Quantifies how surprising the data is under the null hypothesis
- Standardizes hypothesis testing across studies
- Helps control false-positive rates via alpha
- Flags results worth investigating further
AI Mentor Explanation
Suppose you claim a bowler is just average (the null hypothesis). He then takes five wickets in an over-heavy spell. The p-value asks: if he really were average, how likely is a haul this good or better purely by luck? A tiny probability means such a performance is very surprising under the average assumption, so you start doubting he is merely average rather than genuinely skilled.
Step-by-Step Explanation
Step 1
State the hypotheses
Define the null hypothesis (no effect) and the alternative hypothesis you want to test for.
Step 2
Choose alpha
Set the significance level, commonly 0.05, before looking at the data.
Step 3
Compute the test statistic
Calculate the appropriate statistic (t, z, chi-square) from your sample.
Step 4
Find the p-value
Determine the probability of a result at least as extreme as observed under the null hypothesis.
Step 5
Compare and decide
If p is less than or equal to alpha, reject the null; otherwise fail to reject it, and always report effect size.
What Interviewer Expects
- Correct definition conditioned on the null hypothesis being true
- Comparison to a pre-set significance level alpha
- Awareness that p-value is not P(null is true)
- Distinction between statistical and practical significance
- Understanding of one-tailed versus two-tailed interpretation
Common Mistakes
- Interpreting the p-value as the probability the null hypothesis is true
- Treating p = 0.05 as a hard, meaningful boundary
- Confusing statistical significance with a large or important effect
- Choosing alpha after seeing the data (p-hacking)
- Reporting a p-value without an effect size or confidence interval
Best Answer (HR Friendly)
“A p-value tells you how surprising your results would be if there were really nothing going on. A small p-value means your data would be unlikely by chance alone, so you have evidence that something real is happening, though it does not tell you how big or important that something is.”
Code Example
from scipy import stats
group_a = [72, 75, 68, 80, 79, 74, 77]
group_b = [85, 88, 90, 83, 91, 87, 86]
t_stat, p_value = stats.ttest_ind(group_a, group_b)
alpha = 0.05
print(f"t = {t_stat:.3f}, p = {p_value:.4f}")
if p_value <= alpha:
print("Reject the null: the difference is statistically significant.")
else:
print("Fail to reject the null: not enough evidence of a difference.")Follow-up Questions
- What is the difference between statistical and practical significance?
- What is the difference between a one-tailed and a two-tailed test?
- What are Type I and Type II errors in relation to alpha?
- What is p-hacking and how do you guard against it?
- How do confidence intervals complement p-values?
MCQ Practice
1. A p-value is the probability of observing data at least as extreme as yours, assuming what?
The p-value is computed under the assumption that the null hypothesis is true.
2. If p = 0.02 and alpha = 0.05, what is the correct decision?
Since 0.02 is less than or equal to 0.05, you reject the null hypothesis at the 5 percent level.
3. Which statement about p-values is TRUE?
A small p-value means the observed data would be unlikely if the null hypothesis were true; it is not the probability the null is true nor a measure of effect size.
Flash Cards
Define p-value. — Probability of data at least as extreme as observed, given the null hypothesis is true.
What does a small p-value indicate? — The data would be unlikely under the null, giving evidence against it.
Is p-value the probability the null is true? — No. It is conditioned on the null being true; it is not P(null is true).
Statistical vs practical significance? — A significant p-value can still accompany a trivial, unimportant effect size.