liveBook · Manning

Welch's t-test

As explained in Machine Learning System Design

A statistical test for comparing the means of two groups without assuming that their variances are equal.

Welch's t-test compares the means of two groups when their variances may differ. It frames the observed gap as either noise under a null hypothesis or statistically significant evidence against it. section 12.2.4

Use it in an A/B experiment by:

  • Defining the null hypothesis as no difference between group means and the alternative as a non-noise difference. section 12.2.4
  • Interpreting the p-value against a significance level, commonly α = 0.05, to standardize the decision. section 12.2.4
  • Reporting confidence intervals, graphical metric trends, group sizes, absolute totals, methodology, and next steps. section 12.4.1

Retail model experiment

A retail experiment splits representative stores into groups A and B through distribution centers, using one check as the atomic observation. Welch's t-test compares the groups over one month, with a 5% significance level and 10% type II error target; results include revenue, profit, checks, and model metrics. section 12.4.1

How Machine Learning System Design covers it

How deep it goes
Defines the test, hypotheses, p-value, and significance level, but defers detailed guidance on choosing among statistical tests.
Examples
Uses representative retail store groups, daily checks, model metrics, revenue, profit, and a month-long experiment.
What it assumes
Assumes an A/B experiment, two groups, measurable observations, and hypotheses while leaving test-selection guidance outside scope.
What you can do after
Evaluate two-group experiment results and report significance alongside confidence intervals, graphical trends, methodology, and recommendations.
How it connects
Connects A/B testing with Student's t-test, null hypotheses, p-values, confidence intervals, and type I and type II errors.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.