Bayesian Statistics Cheat Sheet
Introduction to Bayesian inference, priors, likelihoods, and posteriors, with practical examples using PyMC for probabilistic modeling.
2 PagesAdvancedMar 2, 2026
Bayes' Theorem by Hand
Manually compute a posterior probability.
python
# Bayes' Theorem: P(A|B) = P(B|A) * P(A) / P(B)# Example: disease testingp_disease = 0.01 # prior: 1% of population has the diseasep_pos_given_disease = 0.95 # test sensitivity (true positive rate)p_pos_given_no_disease = 0.05 # false positive ratep_no_disease = 1 - p_diseasep_positive = (p_pos_given_disease * p_disease + p_pos_given_no_disease * p_no_disease)# Posterior: P(disease | positive test)p_disease_given_pos = (p_pos_given_disease * p_disease) / p_positiveprint(f"P(disease | positive test) = {p_disease_given_pos:.3f}") # ~0.16
Bayesian Model with PyMC
Estimate a coin's bias using MCMC sampling.
python
import pymc as pmimport numpy as npdata = np.array([1, 0, 1, 1, 1, 0, 1, 1, 0, 1]) # coin flips, 1=headswith pm.Model() as model: # Prior belief about the coin's bias theta = pm.Beta("theta", alpha=1, beta=1) # uniform prior # Likelihood of the observed data given theta obs = pm.Bernoulli("obs", p=theta, observed=data) # Sample from the posterior using MCMC trace = pm.sample(2000, tune=1000, return_inferencedata=True)print(pm.summary(trace))
Bayesian Concepts
Core vocabulary of Bayesian inference.
- Prior P(θ)- belief about a parameter before seeing data
- Likelihood P(D|θ)- probability of observing the data given a parameter value
- Posterior P(θ|D)- updated belief about the parameter after observing data; proportional to likelihood times prior
- Evidence P(D)- normalizing constant, probability of the data averaged over all parameter values
- Conjugate prior- a prior that, combined with a given likelihood, yields a posterior in the same family (e.g. Beta-Bernoulli)
- MCMC- Markov Chain Monte Carlo; sampling method to approximate posteriors that lack a closed form
- Credible interval- Bayesian analog of a confidence interval; range containing the parameter with a given posterior probability
Bayesian vs Frequentist
Contrasting the two statistical philosophies.
- Parameters- Bayesian treats parameters as random variables with distributions; frequentist treats them as fixed unknowns
- Uncertainty- Bayesian expresses uncertainty as a probability distribution over parameters; frequentist uses sampling variability
- Prior information- Bayesian formally incorporates prior beliefs; frequentist relies only on the observed data
- Interval interpretation- a 95% credible interval directly means 95% probability the parameter lies within it; a confidence interval does not
- Small samples- Bayesian methods can be more stable with small samples if the prior is reasonable
Closed-Form Conjugate Update
Update a Beta prior with Binomial data analytically, no sampling required.
python
from scipy import stats# Prior belief about a conversion rate: Beta(alpha=2, beta=8) -> mean 0.2, weakly informativealpha_prior, beta_prior = 2, 8# Observed data: 45 conversions out of 300 visitorssuccesses, trials = 45, 300# Conjugacy: posterior is also Beta, with a simple closed-form updatealpha_post = alpha_prior + successesbeta_post = beta_prior + (trials - successes)posterior = stats.beta(alpha_post, beta_post)print(f"posterior mean: {posterior.mean():.4f}")print(f"95% credible interval: {posterior.interval(0.95)}")# Probability the true rate exceeds a business threshold, e.g. 12%p_above_threshold = 1 - posterior.cdf(0.12)print(f"P(rate > 12%) = {p_above_threshold:.3f}")
MCMC Convergence Diagnostics
Never trust a posterior from a sampler you haven't checked for convergence.
python
import arviz as azimport pymc as pmwith pm.Model() as model: theta = pm.Beta("theta", alpha=1, beta=1) pm.Bernoulli("obs", p=theta, observed=[1, 0, 1, 1, 0, 1, 1, 1, 0, 1]) trace = pm.sample(2000, tune=1000, chains=4, random_state=0)# R-hat close to 1.0 (< 1.01) means chains agree; higher means non-convergencesummary = az.summary(trace, var_names=["theta"])print(summary[["r_hat", "ess_bulk", "ess_tail"]])# ess_bulk/ess_tail: effective sample size; low values mean high autocorrelation# Divergences indicate the sampler struggled with the posterior geometryn_divergent = trace.sample_stats.diverging.sum().item()print(f"divergences: {n_divergent}")az.plot_trace(trace, var_names=["theta"]) # visually inspect mixing
Hierarchical (Partial Pooling) Model
Share statistical strength across groups instead of pooling fully or not at all.
python
import pymc as pmimport numpy as np# Conversion counts for 5 stores with different traffic volumestrials = np.array([100, 40, 250, 60, 120])successes = np.array([12, 3, 40, 5, 18])n_groups = len(trials)with pm.Model() as hierarchical_model: # Hyperpriors: shared population-level parameters mu = pm.Beta("mu", alpha=2, beta=8) # population mean rate kappa = pm.Gamma("kappa", alpha=2, beta=0.1) # population concentration # Reparameterize Beta by mean/concentration for interpretability alpha = mu * kappa beta = (1 - mu) * kappa # Each store gets its own rate, pulled toward the population mean theta = pm.Beta("theta", alpha=alpha, beta=beta, shape=n_groups) pm.Binomial("obs", n=trials, p=theta, observed=successes) trace = pm.sample(2000, tune=1000, chains=4, target_accept=0.95, random_state=0)# Small stores' estimates shrink toward mu; large stores stay closer to their own data
Bayesian Model Comparison
Compare models by predictive accuracy instead of raw likelihood.
python
import arviz as azimport pymc as pmdef fit(degree, x, y): with pm.Model() as m: coefs = pm.Normal("coefs", 0, 5, shape=degree + 1) sigma = pm.HalfNormal("sigma", 5) mu = sum(coefs[i] * x**i for i in range(degree + 1)) pm.Normal("y", mu=mu, sigma=sigma, observed=y) idata = pm.sample(1000, tune=1000, random_state=0, idata_kwargs={"log_likelihood": True}) return idatamodels = {f"degree_{d}": fit(d, x_data, y_data) for d in [1, 2, 3]}# LOO (leave-one-out CV, via Pareto-smoothed importance sampling) — preferred over WAICcomparison = az.compare(models, ic="loo")print(comparison) # ranks by expected log predictive density; flags high Pareto-k warnings
Diagnostics & Inference Terms
Vocabulary for evaluating and comparing fitted Bayesian models.
- R-hat (Gelman-Rubin statistic)- ratio of between-chain to within-chain variance; should be < 1.01 for convergence
- Effective sample size (ESS)- number of independent-equivalent samples after accounting for autocorrelation; aim for > 400
- Divergent transitions- NUTS sampler failures indicating regions of extreme posterior curvature; often fixed by reparameterizing or raising target_accept
- WAIC / LOO- information criteria estimating out-of-sample predictive accuracy; lower (or higher ELPD) is better
- Bayes factor- ratio of marginal likelihoods between two models; quantifies relative evidence, distinct from a p-value
- Posterior predictive check- simulate new data from the fitted posterior and compare to observed data to assess model fit
- Variational inference (ADVI)- fast approximate alternative to MCMC that optimizes a simpler distribution to match the posterior
Pro Tip
Always run a prior predictive check before fitting real data — sample from your prior alone and see if it generates plausible values; a prior that puts most of its mass on nonsensical outcomes will bias or destabilize the posterior.
Was this cheat sheet helpful?
Explore Topics
#BayesianStatistics#BayesianStatisticsCheatSheet#DataScience#Advanced#BayesTheoremByHand#BayesianModelWithPyMC#BayesianConcepts#BayesianVsFrequentist#MachineLearning#CheatSheet#SkillVeris
