To test a new recommendation model, compare it with the current model on live platform data. Randomly assign participants to groups so the model is the main difference; compare their outcomes over time. section 4.4
Random assignment helps avoid comparisons skewed by characteristics that affect the outcome:
- In a sunscreen trial, dividing participants by skin tone could distort the results. section 4.4
- In an A/B test, compare recommendation models using selected metrics; whether an increase is desirable depends on what a metric measures. section 4.4
Figure 4.4 Randomized controlled trial in medicine: Patients are randomly split into two groups and, without their knowledge, assigned either the drug or the placebo. The effects on the population are later measured to understand the effects of the drug. In this example, we assume that a population has an ailment for which a new drug has been developed. The drug is given to a random subset of the population, and then the effects are compared with those who didn’t receive the drug. It is vital to do so to understand the baseline of remission. In this case, we can see how the drug is associated with doubling the number of people who are cured.
A drug-versus-placebo trial illustrates random assignment and comparing outcomes against a control group. section 4.4
