Hypothesis testing

As explained in Machine Learning System Design

A statistical procedure for evaluating whether observed data provide evidence against a specified hypothesis or reference distribution.

In an A/B test, a statistical test helps decide whether an observed difference is more than noise. The null hypothesis assumes no difference between group means; the alternative says the difference is statistically significant. A p-value measures how likely the observed statistic, or a more extreme one, would be if the null were true. Compare it with a chosen significance level, often α = 0.05. section 12.2.4

Choose and check the test against the experiment:

• A t-test is common; Welch’s t-test is an option when the two groups’ variances are unequal. section 12.2.4
• Test choice depends on the experiment and data; a detailed selection guide is outside this treatment. section 12.2.4
• Simulate groups at the planned sample size, rerun the test 1,000–10,000 times, and use rejection rates to estimate sensitivity and type II error. section 12.2.5

Figure 12.8 Simulation provides a final validation that everything is correct and set up to run an A/B test.

Use the simulation figure as a final check that the test and sample size are set up appropriately for an A/B test. section 12.2.5

Beyond experiments, statistical tests can help detect changes in model predictions or input-data distributions during monitoring. section 14.4.3

How Machine Learning System Design covers it

How deep it goes

Defines hypotheses, p-values, and significance, then outlines simulation checks; defers detailed guidance on choosing a test.

Examples

Uses unequal-variance group comparisons, repeated simulated experiments, and drift detection as applications.

What it assumes

Assumes an A/B experiment with two groups and planned sample sizes, but defines the null and alternative hypotheses.

What you can do after

Evaluate A/B differences against a null hypothesis and validate test sensitivity through simulation.

How it connects

Connects A/B testing with t-tests, Welch’s t-test, p-values, sensitivity, and type II error.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking StatisticsA statistical procedure for evaluating evidence against a specified null hypothesis about a population or data-generating process.
• 

  liveBookmanning.com
  Causal AIA hypothesis test is a statistical procedure for evaluating whether observed data provide sufficient evidence against a specified null hypothesis.
• 

  liveBookmanning.com
  Statistics Every Programmer NeedsA hypothesis test is a statistical procedure for evaluating evidence against a specified null hypothesis.

Compare how 4 books explain hypothesis testing →

Related concepts

• t-testincludes · A t-test is a statistical hypothesis test that uses a t-distributed statistic to assess a mean or difference in means under specified assumptions.
• chi-square testincludes · A chi-square test is a statistical hypothesis test that uses a chi-square-distributed statistic to evaluate hypotheses about categorical data, goodness of fit, or variance.
• p-valueuses · A p-value is the probability, assuming a specified null hypothesis, of obtaining a test statistic or result at least as extreme as the observed one.
• null hypothesisuses · A null hypothesis is a designated baseline statistical claim, typically asserting that an effect, difference, or association is absent.
• one-sided testhas part · A one-sided test is a hypothesis test whose alternative hypothesis and rejection region concern a departure from the null in one prespecified direction.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.