A treatment variant is an alternative model configuration tested against a control or other variants. It may change model versions, feature weights, prompts, retrieval strategies, or post-processing rules. section 6.5.1
Use treatment variants to turn a modeling change into an interpretable product experiment:
- Attach a clear hypothesis to every variant, including the expected behavior difference and how conflicting or similar results will be interpreted. section 6.5.1
- Select variants with a playbook that checks product goals, stakeholder requirements, model quality, and operational readiness before consuming A/B-test capacity. section 7.1
- Limit unnecessary branches: additional variants require more traffic or duration and increase false-positive risk through multiple comparisons. section 6.5.1
Movie recommendation treatments
A control uses the current recommendation model. Treatment A adds contextual features with conservative weighting, while Treatment B gives stronger weight to recent searches and session context. The test asks whether contextual features improve recommendations and how much weight they should receive. User-level randomization, defined exposure rules, a precomputed sample size, primary metrics, and guardrails make the comparison interpretable. section 6.3.6
Interpret a variant through product outcomes rather than a single metric: a primary-metric gain must be weighed against guardrail degradation, added complexity, serving cost, and operational risk. section 6.3.6 section 6.5.1
