liveBook · Manning

Variant

As explained in AI Model Evaluation

A test variant, or treatment, is an experimental condition that receives the change being evaluated.

A variant is an experimental condition in an A/B test. The test variant, or treatment, exposes selected users to a change; the control variant provides the unchanged product experience. Comparing them helps measure the change’s impact on users and product outcomes. section 6.3.2 section 1.6

Define variants alongside the randomization unit, success metrics, guardrail metrics, and any planned segments. Measure each variant consistently: a model may improve its primary success metric while harming retention or creating bias, so guardrails matter too. section 6.3.2 section 6.3.5

Assignment is not the same as exposure. A user assigned to treatment may never reach the relevant product surface, or may see a fallback or cached result instead of the new model output. Log both assignment and exposure, and record the model version and relevant serving behavior before interpreting differences. section 6.3.5

Checking a 50/50 test

If a model test plans an equal split between control and treatment, check that the observed traffic is roughly 50/50. Then verify randomization, exposure logging, fallback behavior, overlapping experiments, and consistent metric calculations. A dashboard alone does not establish that the comparison is valid. section 6.3.5

Model tests may include several test variants, such as versions with different feature weights. More variants increase configuration complexity and can require larger samples or longer tests. Offline evaluations can help select which variants to test online, but cannot replace measuring their impact on real users. section 6.8 section 1.6

How AI Model Evaluation covers it

How deep it goes
Defines variants and gives validity checks for assignment, exposure, allocation, logging, and metrics, without providing an implementation.
Examples
Illustrates a 50/50 control-treatment split and model variants with different feature weights.
What you can do after
Evaluate experiment validity by checking variant allocation, exposure, serving behavior, and metric consistency before interpreting results.
How it connects
Connects variants to randomization, success metrics, guardrail metrics, segmentation, and offline evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.