Hypothesis testing is a statistical experiment for asking whether sample data gives strong reason to challenge a population claim. Start with a null hypothesis (H0), the status-quo assumption, and an alternative hypothesis (Ha), the competing claim. A soda can labeled 305 mL, for example, prompts H0: its population mean is 305 mL; Ha asks whether it differs enough to question that label. section 6.0 section 6.15
The key question is not whether a sample differs exactly from H0—samples nearly always vary—but whether it is unusually different if H0 were true. Calculate a test statistic from the sample, then compare its p-value with a chosen significance level, α. Reject H0 when the statistic lands in the critical region or when p < α; otherwise, the data has not supplied enough evidence to reject it. section 6.15 section 6.0
Choose the test shape from the claim before looking at the result:
- Use a two-tailed test for any difference, such as Ha: mean ≠ 305 mL. Split α across both tails. section 6.15
- Use a left-tailed test for a “less than” claim or a right-tailed test for a “greater than” claim. Keep α in the relevant tail. section 6.15
- For means with samples of 30 or fewer, use a t-distribution with degrees of freedom n − 1; use the standard normal distribution for n > 30, while recognizing that t is often preferred in practice. section 6.12 section 6.15
Could tea identification be luck?
Eight cups contain four milk-first and four tea-first preparations. If someone guessing at random identifies every cup correctly, simulation estimates that outcome at about 1.4% of trials. That is a p-value: the probability of observing this outcome if the null hypothesis—random guessing—is true. It is not the probability that the person was guessing after seeing the outcome; that would be a posterior question deferred to Bayesian statistics. section 6.1
import random
n_trials=1_000_000 #1
succeeded = 0 #2
for _ in range(n_trials): #3
cups = list(range(8)) #4
true_milk_first = set(random.sample(cups, 4)) #5
guess_milk_first = set(random.sample(cups, 4)) #6
correct = len(true_milk_first & guess_milk_first) #7
if correct == 4: #8
succeeded += 1 #8
print(f"\nProportion of outcomes with all 4 correct: {succeeded /
n_trials:.4f}") #9
Run the random simulation to see how repeated guessing produces a small fraction of perfect eight-cup results, making the p-value tangible rather than mystical. section 6.1
A test weighs evidence rather than proving a claim true or false forever. Results deserve humility: data can be biased, samples can be weak, and researchers can face incentives that favor preferred outcomes. Use the result as a reason to update a claim and remain open to later evidence. section 6.2 section 7.2
