liveBook · Manning

Hypothesis testing

As explained in Grokking Bayes

A hypothesis test is a statistical procedure that evaluates whether observed data provide sufficient evidence against a specified null hypothesis.

Hypothesis testing evaluates whether observed data provide convincing evidence for a claim. A classical test starts with a null hypothesis, models data under that assumption, and measures how unusual the observed result would be. A Bayesian framing instead compares competing models and can connect evidence to decisions. section 11.0 section 11.3

A traditional hypothesis test follows these steps:

  • Specify a null hypothesis that defines the baseline model. section 11.3
  • Choose a test statistic that summarizes the relevant data. section 11.3
  • Calculate how often the null model produces a result at least as extreme as the observation; this probability is the p-value. section 11.2 section 11.3
  • Reject the null when the p-value falls below a chosen significance level, often 0.05. section 11.3

Testing a website redesign

Suppose 63 of 500 visitors using a new design convert, an observed rate of 12.6%. Set the baseline conversion rate to 10% and test whether the redesign performs better. Under the null hypothesis, conversions follow a binomial distribution with probability 0.1. The test asks whether 63 or more conversions would be unusually high under that model. section 11.2

𝐻₀: 𝑟 = 0.1.

Use the compact equation to inspect the null hypothesis for the conversion-rate example. section 11.2

N = 500
n_conversions = 63
baseline = 0.1
 
test = binomtest(
    n_conversions,         #A
    N,                     #B
    baseline,              #C
    alternative="greater"  #D
)
p_value = test.pvalue

Use the Python listing to translate the binomial test into a p-value calculation. section 11.2

How Grokking Bayes covers it

How deep it goes
Explains null hypotheses, test statistics, tail probabilities, significance thresholds, binomial tests, and two-sample t-tests.
Examples
Uses a website redesign, 500 visitors, 63 conversions, a 10% baseline, and a Python binomial-test listing.
What it assumes
Uses basic probability and binomial distributions while defining the binomial model through the conversion example.
What you can do after
Apply a binomial test, compute a p-value, and make a threshold-based decision about evidence against a null hypothesis.
How it connects
Connects null hypothesis, p-value, binomial distribution, t-test, Bayesian statistics, Bayes factor, and decision-making.

Also explained in

Compare how 8 books explain hypothesis testing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.