What is a Hypothesis Test?
Learn what a hypothesis test is, the null and alternative hypothesis, Type I and Type II errors, statistical power, and how to run one in Python.
Expected Interview Answer
A hypothesis test is a formal statistical procedure for deciding, using sample data, whether there is enough evidence to reject a null hypothesis of no effect or no difference in favor of an alternative hypothesis, by comparing a computed p-value against a pre-chosen significance level such as 0.05.
The standard procedure states a null and alternative hypothesis, fixes a significance level (alpha) before collecting data, selects an appropriate test such as a t-test for comparing means, a chi-square test for categorical association, or ANOVA for comparing more than two group means, then computes a test statistic and its p-value from the sample. A Type I error is rejecting a true null hypothesis, a false positive, controlled by alpha; a Type II error is failing to reject a false null hypothesis, a false negative, whose probability is reduced by increasing statistical power, often via a larger sample size. Correct interpretation requires saying you 'fail to reject' the null rather than 'accept' it, since absence of evidence is not evidence of absence. Running many tests without correction, such as the Bonferroni method, inflates the chance of false positives across the set of tests.
- Provides a disciplined, repeatable framework for evaluating evidence
- Controls the false-positive rate explicitly via alpha
- Applies across many data types via different tests (t-test, chi-square, ANOVA)
- Forces pre-registration of decision criteria before data collection
- Underpins rigorous A/B testing and experimental design
AI Mentor Explanation
A hypothesis test is like a selector deciding whether a new training method really improved a batter's average, starting from the skeptical assumption of no real improvement and only overturning it if the evidence is strong enough. They fix that threshold before the trial starts, then compare pre- and post-training scores, accepting a small risk of crediting a method that did nothing.
Step-by-Step Explanation
Step 1
State the hypotheses
Write a precise null hypothesis (no effect) and alternative hypothesis (an effect exists).
Step 2
Choose the significance level
Fix alpha, commonly 0.05, before collecting or looking at the data.
Step 3
Select the appropriate test
Pick a t-test, chi-square test, ANOVA, or another test based on the data type and design.
Step 4
Compute the test statistic and p-value
Run the test on your sample data to get a p-value.
Step 5
Make a decision
Reject the null hypothesis if p is less than alpha; otherwise, fail to reject it.
Step 6
Consider errors and power
Weigh the risk of a Type I error against a Type II error, and check the test has enough statistical power to detect a real effect.
What Interviewer Expects
- Walks through the full hypothesis-testing procedure in order
- Correctly defines Type I and Type II errors
- Explains statistical power and why sample size matters
- Names at least one appropriate test for a given data scenario
- Is careful about interpreting 'fail to reject' versus 'accept' the null
Common Mistakes
- Saying you 'accept' the null hypothesis instead of 'fail to reject' it
- Choosing alpha after seeing the p-value
- Confusing Type I and Type II errors
- Running many tests without correcting for multiple comparisons
- Ignoring statistical power and sample size when planning the test
Best Answer (HR Friendly)
“A hypothesis test is a structured way to check, using data, whether something you observed is likely a real effect or just random noise. You start by assuming there's no real effect, then see if the evidence is strong enough to change your mind, while accepting a small, known risk of being wrong.”
Code Example
from scipy import stats
control = [3.1, 3.4, 2.9, 3.2, 3.0, 3.3]
treatment = [3.6, 3.9, 3.5, 3.8, 3.7, 4.0]
alpha = 0.05
t_stat, p_value = stats.ttest_ind(control, treatment)
print(f"t-statistic: {t_stat:.3f}, p-value: {p_value:.4f}")
if p_value < alpha:
print("Reject H0: treatment likely has a real effect")
else:
print("Fail to reject H0: not enough evidence of an effect")Follow-up Questions
- What is the difference between a Type I and a Type II error?
- What is statistical power and how do you increase it?
- When would you use a t-test versus a chi-square test?
- What does it mean to 'fail to reject' the null hypothesis, versus 'accept' it?
- How do you handle multiple hypothesis tests run at the same time?
MCQ Practice
1. What is a Type I error in hypothesis testing?
A Type I error occurs when you reject a null hypothesis that is actually true, producing a false positive result.
2. What should you decide before collecting data in a hypothesis test?
Alpha, the significance threshold, must be set before seeing the data to keep the decision process unbiased.
3. What does statistical power measure?
Statistical power is the probability of detecting a real effect when one truly exists, equal to 1 minus the Type II error rate.
Flash Cards
What is the null hypothesis? — The default assumption of no effect or no difference, which the test tries to find evidence against.
What is a Type I error? — Rejecting a true null hypothesis — a false positive.
What is a Type II error? — Failing to reject a false null hypothesis — a false negative.
What is statistical power? — The probability of correctly detecting a real effect when one exists (1 minus the Type II error rate).