What is a P-Value?
Learn what a p-value really means, how it's calculated from a hypothesis test, common misinterpretations, and how to interpret it correctly in interviews.
Expected Interview Answer
A p-value is the probability, assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one actually observed in your sample. A small p-value, typically below a pre-chosen significance level like 0.05, is evidence against the null hypothesis, not proof it is false.
P-values come from a specific test (t-test, chi-square, ANOVA, etc.) that produces a test statistic, which is then converted into a probability under the assumption that there is no real effect. A p-value of 0.03 means that, if there truly were no effect, a result this extreme would happen about 3% of the time by chance alone. It is not the probability the null hypothesis is true, and it is not the probability the result happened by chance; those are common and important misinterpretations. The significance level (alpha), usually 0.05, must be chosen before looking at the data to keep the Type I error rate under control, and a small p-value does not by itself indicate a large or practically important effect.
- Gives a standardized way to judge evidence against a null hypothesis
- Supports data-driven decisions instead of eyeballing results
- Works across many types of tests (t-test, chi-square, ANOVA)
- Forces explicit, pre-registered decision thresholds
- Flags results likely to be random noise versus real signal
AI Mentor Explanation
A p-value is like asking: if a bowler really has no unusual skill, how surprising would it be to see them take six wickets in a match purely by chance? A p-value of 0.02 means that under a normal, skill-free bowler, a haul this extreme would happen only about 2% of the time, strong evidence something more than luck is going on, not proof of it.
Step-by-Step Explanation
Step 1
State the hypotheses
Define a null hypothesis (no effect or difference) and an alternative hypothesis.
Step 2
Choose a significance level
Pick alpha, commonly 0.05, before looking at the data, as the threshold for rejecting the null.
Step 3
Compute the test statistic
Run the appropriate test (t-test, chi-square, etc.) on your sample data.
Step 4
Calculate the p-value
Find the probability of a result this extreme or more, assuming the null hypothesis is true.
Step 5
Compare to alpha
If p is less than alpha, reject the null hypothesis; otherwise, fail to reject it.
Step 6
Interpret carefully
State that you found, or didn't find, evidence against the null, never that you 'proved' the alternative.
What Interviewer Expects
- Gives the correct formal definition of a p-value
- Explicitly rejects the 'probability null is true' misinterpretation
- Connects the p-value to significance level and Type I error
- Distinguishes statistical significance from practical significance
- Can walk through a concrete example
Common Mistakes
- Saying a p-value is the probability the null hypothesis is true
- Saying a p-value is the probability the result happened by chance
- Treating p less than 0.05 as proof of a large or important effect
- Choosing alpha after seeing the p-value (p-hacking)
Best Answer (HR Friendly)
“A p-value tells you how surprising your data would be if there were actually no real effect. A small p-value, like 0.02, means the result would be unlikely to happen by chance alone, giving you evidence that something real is going on, though it's not absolute proof.”
Code Example
from scipy import stats
group_a = [5.1, 4.9, 5.3, 5.0, 4.8]
group_b = [5.6, 5.4, 5.8, 5.5, 5.7]
t_stat, p_value = stats.ttest_ind(group_a, group_b)
print(f"p-value: {p_value:.4f}")
alpha = 0.05
if p_value < alpha:
print("Reject the null hypothesis: groups likely differ")
else:
print("Fail to reject the null hypothesis")Follow-up Questions
- What is the difference between a p-value and a significance level (alpha)?
- What is a Type I error versus a Type II error?
- Why doesn't a p-value tell you the probability the null hypothesis is true?
- What is p-hacking and how do you avoid it?
- How does sample size affect the p-value?
MCQ Practice
1. What does a p-value represent?
A p-value is calculated under the assumption that the null hypothesis holds, not as a probability about the hypotheses themselves.
2. If p equals 0.03 and alpha equals 0.05, what do you conclude?
Since 0.03 is less than the 0.05 threshold, the result is considered statistically significant and the null is rejected.
3. Which statement about p-values is correct?
A small p-value is evidence against the null hypothesis; it never constitutes proof, and it does not measure effect size.
Flash Cards
What is a p-value, formally? — The probability of a result at least as extreme as observed, assuming the null hypothesis is true.
Does a p-value tell you the null hypothesis is true? — No — it never gives the probability that the null (or alternative) hypothesis is true.
What is the common significance threshold? — 0.05, though it should be chosen based on context, not by default.
What is p-hacking? — Manipulating an analysis, such as trying many tests, until a p-value under 0.05 appears, inflating false positives.