liveBook · Manning

Proportional z-test

As explained in Machine Learning System Design

A statistical test for comparing proportions between groups using a standardized normal-test statistic.

Use a proportional z-test when an experiment compares rates such as click-through rate (CTR) or conversion rate (CVR) between groups. section 12.4.2

A practical testing workflow:

  • Set conservative error rates; the example uses α = 0.01 and β = 0.05. section 12.4.2
  • Estimate baseline and target proportions before calculating the required sample size. section 12.4.2
  • Compare the required observations with expected daily traffic to decide experiment duration. section 12.4.2

PhotoStock search experiment

For a search-engine A/B test, a 10% baseline CTR and 10.5% target CTR require 131,095 views per group. With 150,000 daily views per group, the planned detectable increase needs one day of data. section 12.4.2

How Machine Learning System Design covers it

How deep it goes
Explains metric selection, error-rate choices, baseline calculations, and sample-size planning without deriving the test statistic.
Examples
Uses PhotoStock search traffic, clicks, purchases, baseline rates, group allocation, and a one-day sample-size calculation.
What it assumes
Assumes familiarity with proportions, CTR, CVR, A/B groups, and type I and II errors.
What you can do after
Evaluate proportion differences and estimate the observation time needed to detect a specified CTR or CVR increase.
How it connects
Connects A/B testing with CTR, CVR, sample-size estimation, minimum detectable effect, and type I and II errors.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.