A variant is an experimental condition in an A/B test. The test variant, or treatment, exposes selected users to a change; the control variant provides the unchanged product experience. Comparing them helps measure the change’s impact on users and product outcomes. section 6.3.2 section 1.6
Define variants alongside the randomization unit, success metrics, guardrail metrics, and any planned segments. Measure each variant consistently: a model may improve its primary success metric while harming retention or creating bias, so guardrails matter too. section 6.3.2 section 6.3.5
Assignment is not the same as exposure. A user assigned to treatment may never reach the relevant product surface, or may see a fallback or cached result instead of the new model output. Log both assignment and exposure, and record the model version and relevant serving behavior before interpreting differences. section 6.3.5
Checking a 50/50 test
If a model test plans an equal split between control and treatment, check that the observed traffic is roughly 50/50. Then verify randomization, exposure logging, fallback behavior, overlapping experiments, and consistent metric calculations. A dashboard alone does not establish that the comparison is valid. section 6.3.5
Model tests may include several test variants, such as versions with different feature weights. More variants increase configuration complexity and can require larger samples or longer tests. Offline evaluations can help select which variants to test online, but cannot replace measuring their impact on real users. section 6.8 section 1.6
