A/B testing compares ranking alternatives with live or simulated users. Random assignment sends users to different variants, while tracked outcomes such as purchases, sales, time spent, or retention reveal which variant performs better. section 12.0 section 12.2.2
Search experiments commonly use the current ranking algorithm as a control and compare it with a candidate learning-to-rank model. This tests whether offline model quality survives real-world behavior rather than treating laboratory performance as conclusive. section 12.2.2 section 12.2
A practical search A/B test follows this pattern:
- Randomly assign each user to one of two ranking variants. section 12.2.2
- Serve the assigned model and observe interactions such as clicks and purchases. section 12.2.3
- Aggregate the selected business outcome for each variant and compare performance. section 12.0 section 12.2.3
- Use the result to correct offline training problems and improve the retraining loop. section 12.2.2
A simulated search experiment
For the query "transformers dvd", randomly assign 1,000 simulated users to two ranking models. Each session may produce clicks and a purchase; aggregate purchases by model. One run returns 21 purchases for ltr_model_variant_1 and 15 for ltr_model_variant_2, despite the second model appearing stronger in offline tests. section 12.2.3
Figure 12.3 A search A/B test. Search users are randomly assigned to two relevance algorithms (here two LTR models) with outcomes tracked.
Use the search A/B-test diagram to trace random assignment from users to alternative relevance algorithms and tracked outcomes. section 12.2.2
