liveBook · Manning

Randomized experiment

As explained in Hidden Influences

An experiment that randomly assigns participants to treatment and control groups to estimate the effect of an intervention.

To test a new recommendation model, compare it with the current model on live platform data. Randomly assign participants to groups so the model is the main difference; compare their outcomes over time. section 4.4

Random assignment helps avoid comparisons skewed by characteristics that affect the outcome:

  • In a sunscreen trial, dividing participants by skin tone could distort the results. section 4.4
  • In an A/B test, compare recommendation models using selected metrics; whether an increase is desirable depends on what a metric measures. section 4.4
Figure 4.4 Randomized controlled trial in medicine: Patients are randomly split into two groups and, without their knowledge, assigned either the drug or the placebo. The effects on the population are later measured to understand the effects of the drug. In this example, we assume that a population has an ailment for which a new drug has been developed. The drug is given to a random subset of the population, and then the effects are compared with those who didn’t receive the drug. It is vital to do so to understand the baseline of remission. In this case, we can see how the drug is associated with doubling the number of people who are cured.

A drug-versus-placebo trial illustrates random assignment and comparing outcomes against a control group. section 4.4

How Hidden Influences covers it

How deep it goes
Explains random assignment and control groups through medical trials, then adapts the comparison to recommendation models.
Examples
Uses a sunscreen trial confounded by skin tone, a drug-versus-placebo trial, and competing recommendation models.
What you can do after
Evaluate recommendation models by comparing selected measurable outcomes against a baseline in a live A/B test.
How it connects
Connects randomized controlled trials, A/B testing, recommender systems, and outcome metrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Hidden Influences. The section links point to the parts of the book it draws on.