p-values, Significance Level and Type I/II Errors — Statistics & Probability for Data Science

The p-value is the most used and most misunderstood number in all of statistics, and the errors a hypothesis test can make — false alarms and missed effects — determine the real-world consequences of every data-driven decision. This lesson exists to clarify exactly what a p-value is, what the significance level controls, and how the two types of error trade off against each other. Without a precise grasp of these concepts, analysts misinterpret p-values as the probability the null is true, set thresholds without understanding the error rates they imply, and remain blind to the missed-effect errors that never announce themselves. Mastering this material is what separates someone who runs tests mechanically from someone who understands the risks those tests are actually managing.

Analogy🏏Cricket

🏏 Think of it like cricket: Imagine Virat Kohli has scored 45, 52, 38, 61, and 49 across five innings in a series, and a commentator wants to describe his form in one phrase. The commentator cannot read out all five scores every time, so they compress them into a single representative figure. Just as the commentator picks one number to stand in for the whole sequence of innings, a measure of central tendency picks one value to represent an entire dataset. Just as a misleading summary ('he averages 200') would distort how selectors judge Kohli, a wrongly chosen centre distorts how analysts judge data. And just as different summaries (best score versus typical score) tell different stories, the mean, median, and mode each emphasise a different aspect of the same innings. This reveals why central tendency is never one fixed number — it is a deliberate choice about which story the data should tell.