A multi-armed bandit (MAB) is an adaptive way to test alternatives while balancing exploration—learning which option works—and exploitation—using the option currently producing the best reward. It can evaluate models with real users in production or in a controlled test, often gathering useful evidence more efficiently than a fixed A/B test. section 8.2.6 section 8.2.5
Use a bandit evaluation to make model testing an explicit operational decision:
- Define the reward signal before deployment. Explicit feedback can include a recommended item being purchased; implicit feedback can also indicate model performance. section 8.2.6
- Allocate trials among candidate models, then reduce exposure to an option that repeatedly produces weak rewards instead of continuing to spend live traffic on it. section 8.2.6
- Record the performance measures, costs, trade-offs, and decision basis. Live testing can expose people to experimental behavior, so evaluation constraints belong in the selection record. section 8.6
- Use the approach when rapid learning matters more than the slower, statistically strong evidence associated with a conventional A/B test. section 8.2.5
Three machines, six pulls
Three machines represent candidate processes: A pays frequently, B pays occasionally, and C never pays. After C fails to reward two pulls, attention shifts to A and B. By the sixth pull, B has failed four times out of six while A has failed twice, so trials stop favoring B; A continues paying. section 8.2.6
Figure 8.2 Three one-arm bandits with pay-out records, where each block represents one pull. A light record indicates a zero reward, and a dark one indicates a positive reward. Machine A pays frequently, Machine B pays occasionally, and Machine C never pays.
Inspect the payout record to see how repeated rewards and failures guide the balance between exploration and exploitation. section 8.2.6
