Run Hypothesis Tests and Interpret p-values — Statistics & Probability for Data Science

What You'll Build

In this second stage of the capstone you will build the inferential heart of the analysis, running the appropriate hypothesis tests to determine whether the training regime genuinely works. You will check assumptions, run a paired test on the within-player improvement, a two-group test comparing treatment against control, and a chi-square test on whether injury is associated with group, reporting effect sizes and interpreting every p-value correctly. This stage exists because it answers the central question the whole trial was designed to address — does the regime have a real effect? — and it demands the disciplined test-selection and interpretation skills from Modules 4 and 5. By the end you will have rigorous, assumption-checked verdicts on the regime's effects.

Analogy🏏Cricket

🏏 Think of it like cricket: Building a Monte Carlo simulator is like a team running thousands of practice scenarios in the nets to estimate how often a tactic works, rather than relying on a coach's theoretical hunch. Just as repeatedly playing out the scenario and counting successes reveals the true odds far more reliably than armchair reasoning, simulating thousands of random matches reveals probabilities the maths might struggle to express. Just as more net sessions give a sharper estimate, more simulated trials give a more accurate probability. The insight is that when a probability is hard to derive, you can let randomness itself compute the answer by playing the scenario out enough times to see the pattern emerge.

Prerequisites

Python 3.10 or later with NumPy, SciPy, and pandas installed.
The capstone dataset and the descriptive findings from Lesson 32.
Mastery of hypothesis testing, p-values, and error types from Module 4.
Command of t-tests, chi-square, and non-parametric tests from Module 5.
Understanding of effect sizes and the significance-versus-importance distinction.

Setup & Project Structure

You will create a hypothesis-testing module in the capstone directory that loads the dataset and the improvement column, then runs the suite of tests. Keeping the inference stage separate from the descriptive stage maintains the clean, reviewable structure of the growing analysis. The descriptive findings from the previous stage — particularly the skewness of fitness — directly inform the assumption checks and test choices here.

Analogy🏏Cricket

🏏 Think of it like cricket: setting up this simulation project is like building a reliable practice net before running match scenarios. You create one simulator module — the 'random delivery engine' that bowls virtual balls — the way a bowling machine can reproduce any spell on demand, and you keep the experiments script separate, like the coach's session plan that decides which drills to run today. Just as one machine serves countless practice plans, separating the engine from the experiments lets you reuse the same simulator for many different probability questions. Crucially, just as a bowling machine set to identical settings must deliver the identical sequence so two players face a fair, repeatable test, you seed the random number generator so results are deterministic across runs — essential for debugging and for anyone reproducing your work. The payoff: a clean, seeded structure means your Monte Carlo experiments are trustworthy and repeatable, not a different answer every time you press start.

bash

# In the capstone project directory
touch hypothesis_tests.py

# Confirm prior stage available
python -c "import pandas as pd; df = pd.read_csv('cricket_trial.csv'); \
print('improvement range:', (df.strike_after - df.strike_before).min().round(1), \
'to', (df.strike_after - df.strike_before).max().round(1))"

Step 1 — Foundation

Step 1 builds the assumption-checking foundation that governs every test choice: functions to test normality and equal variance, applied to the variables before any test is run. This is the foundation because, per the standards from Module 4, no test may be run without first verifying its assumptions, and the results here determine whether parametric or non-parametric tests are used. Checking assumptions first embodies the disciplined workflow that separates rigorous from naive analysis.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 1 is like building a perfectly weighted bowling machine that delivers each type of ball with exactly the right frequency — dots most often, wickets rarely. Just as the machine's weighting must faithfully match the real probabilities for the practice to be meaningful, your sampler's probability vector must faithfully encode the delivery distribution. Just as one bowled ball is the atom of an over, one sampled outcome is the atom of every simulation. The insight is that a faithful single-event sampler is the engine from which all larger probabilistic scenarios are assembled.

python

# hypothesis_tests.py
import numpy as np
import pandas as pd
from scipy import stats

trial = pd.read_csv("cricket_trial.csv")
trial["improvement"] = trial["strike_after"] - trial["strike_before"]

def check_normality(sample):
    if len(sample) < 3:
        return False
    return stats.shapiro(sample)[1] > 0.05

def check_equal_var(a, b):
    return stats.levene(a, b)[1] > 0.05

treat = trial[trial.group == "treatment"]
ctrl = trial[trial.group == "control"]
print(f"Treatment improvement normal? {check_normality(treat['improvement'])}")
print(f"Control improvement normal? {check_normality(ctrl['improvement'])}")
print(f"Equal variances? {check_equal_var(treat['improvement'], ctrl['improvement'])}")

Step 2 — Core Logic

Step 2 runs the core hypothesis tests with effect sizes: a paired test comparing each player's after against before to confirm within-player improvement, and a two-group test comparing the treatment group's improvement against the control's to isolate the regime's effect beyond any general change. This is the analytical heart because these two tests directly answer whether the regime works, and computing Cohen's d alongside each ensures the practical magnitude is reported, not just statistical significance.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 2 is like bowling ten thousand practice overs and tallying how many were maidens to estimate your real maiden-over rate. Just as the proportion of maidens across thousands of overs converges to your true maiden probability, the fraction of successful trials in a Monte Carlo run converges to the true event probability. Just as ten overs would give a noisy estimate but ten thousand a reliable one, small simulations are noisy while large ones are precise. The insight is that empirical frequency over many independent trials is a direct, trustworthy estimate of probability.

python

# hypothesis_tests.py (continued)
def cohens_d(a, b):
    pooled = np.sqrt((np.var(a, ddof=1) + np.var(b, ddof=1)) / 2)
    return (np.mean(a) - np.mean(b)) / pooled

# PAIRED test: did players improve vs their own baseline (treatment group)?
t_paired, p_paired = stats.ttest_rel(treat["strike_after"], treat["strike_before"])
print(f"Paired (treatment within-player): t={t_paired:.2f}, p={p_paired:.4g}")

# TWO-GROUP test: treatment improvement vs control improvement (Welch)
t_grp, p_grp = stats.ttest_ind(treat["improvement"], ctrl["improvement"],
                               equal_var=False)
d_grp = cohens_d(treat["improvement"].values, ctrl["improvement"].values)
print(f"Two-group (treatment vs control): t={t_grp:.2f}, p={p_grp:.4g}, "
      f"Cohen's d={d_grp:.2f}")
print(f"  Practical: {'meaningful' if abs(d_grp) >= 0.5 else 'modest'} effect size")

Step 3 — Integration & Enhancement

Step 3 adds the categorical analysis and a power check, completing the inferential suite. You will run a chi-square test on whether injury occurrence is associated with treatment group, applying Module 5's categorical method, and perform a power analysis confirming the study was adequately sized to detect the effects found. This integration ensures the analysis covers every variable type and verifies that any non-significant findings reflect true absence rather than insufficient power, meeting the capstone's rigour standards.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 3 is like watching a new bowler over many overs and continuously revising your estimate of how good he really is, starting from a cautious prior and sharpening it with each wicket or wicketless over. Just as your belief about his true skill tightens as evidence accumulates, the Bayesian posterior concentrates as more simulated deliveries are observed. Just as one good over barely moves your judgement but fifty overs settle it, each observation nudges the posterior proportionally. The insight is that simulation and Bayesian updating combine naturally: simulation produces evidence, and Bayes turns that evidence into refined belief.

python

# hypothesis_tests.py (continued)
from statsmodels.stats.power import TTestIndPower

# CHI-SQUARE: is injury associated with treatment group?
contingency = pd.crosstab(trial["group"], trial["injured"]).values
chi2, p_chi2, dof, _ = stats.chi2_contingency(contingency)
print(f"Chi-square (group vs injury): chi2={chi2:.2f}, p={p_chi2:.4f}")
print(f"  Injury {'associated with' if p_chi2 < 0.05 else 'independent of'} group")

# POWER CHECK: was the study large enough to detect the observed effect?
analysis = TTestIndPower()
d_obs = abs(cohens_d(treat["improvement"].values, ctrl["improvement"].values))
power = analysis.power(effect_size=d_obs, nobs1=len(treat), alpha=0.05)
print(f"Achieved power for the group comparison: {power:.3f}")
print(f"  {'Adequately powered' if power >= 0.8 else 'Underpowered'}")

Step 4 — Testing & Verification

Now you will run the full inferential suite and verify the verdicts match the dataset's known construction. Confirm the paired test detects within-player improvement, the two-group test finds the treatment effect significant with a meaningful effect size, the chi-square reflects the built-in injury-rate difference, and the power check confirms adequacy. Each result recovering the embedded ground truth verifies your inference is correct.

Analogy🏏Cricket

🏏 Think of it like cricket: verifying a Monte Carlo simulation is like trusting a batting average only after enough innings. Estimate the chance of a maiden over from ten simulated overs and the number jumps around wildly, just as a batter's average after two innings tells you little; run ten thousand overs and it settles onto the true value, exactly as an average stabilises over a long career. Just as facing more deliveries lets you read a bowler's real wicket-taking rate rather than being fooled by a lucky first spell, your Bayesian posterior mean creeps toward the true hidden wicket rate as more balls are observed. Convergence toward the known expected answer is your proof the engine and the inference are wired correctly, the way a scorer trusts the system once it reproduces a result everyone already agrees on. The payoff: watching estimates lock onto known truths as trials grow is the trust-check that lets you believe the simulator on questions you can't solve by hand.

bash

# Run the full inferential analysis
from hypothesis_tests import (treat, ctrl, trial, cohens_d, stats)
from statsmodels.stats.power import TTestIndPower
import pandas as pd, numpy as np

t_grp, p_grp = stats.ttest_ind(treat["improvement"], ctrl["improvement"],
                               equal_var=False)
d = cohens_d(treat["improvement"].values, ctrl["improvement"].values)
print(f"Treatment vs control: p={p_grp:.4g}, Cohen's d={d:.2f}")

# CORRECT interpretation reminder:
verdict = "significant" if p_grp < 0.05 else "not significant"
print(f"  Interpretation: IF the regime had no effect, data this extreme would")
print(f"  occur with probability {p_grp:.4g}. Result is {verdict}.")
print(f"  This is NOT 'the probability the regime does nothing'.")

# Expected: significant (p small), d ~ 0.5-1.0 (meaningful), adequately powered.

Warning: When reporting your p-values, never describe them as the probability that the regime has no effect or that the result is due to chance. A p-value is the probability of data at least as extreme as observed, assuming the regime has no effect. Reversing this — the most common error in applied statistics — would misrepresent your central finding to whoever reads the analysis, so state the interpretation in the correct direction every time.

Extension Challenge: Extend the inferential stage to run the analysis non-parametrically as a robustness check — use the Wilcoxon signed-rank test for the paired comparison and the Mann-Whitney test for the two-group comparison — and confirm the conclusions hold regardless of distributional assumptions. As a stretch goal, apply a multiple-comparisons correction across all the tests you ran, since running several tests inflates the family-wise error rate.

The inferential stage answers the trial's central question: does the regime genuinely work?
Assumption checks for normality and equal variance must precede every test, determining parametric versus non-parametric choice.
A paired test confirms within-player improvement; a two-group test isolates the regime's effect beyond general change.
Effect sizes like Cohen's d accompany every test so practical magnitude is reported, not just significance.
A chi-square test addresses the categorical injury outcome, covering every variable type.
A power check confirms the study was large enough, so non-significant results reflect true absence not insufficient power.
Always interpret p-values in the correct direction: the probability of the data under the null, never the probability the null is true.