An arm is one selectable action or candidate system version in a multi-armed bandit. Each arm produces an observable outcome, such as a click, that contributes to a business metric. The current version is an arm too; no arm receives automatic preference. section 3.2
Bandit logic balances exploration—measuring less-known arms—with exploitation—selecting the arm whose realized metric is currently strongest. With epsilon-greedy, exploration selects among K arms at random, while exploitation chooses the arm with the highest observed mean metric. section 3.2 section 3.4
Four click-model arms
An ad selector can evaluate four click-probability models as arms indexed 0 through 3. A simulator assigns CTRs of 0.005, 0.007, 0.009, and 0.011; epsilon-greedy increasingly selects arm 3 and achieves an overall CTR near 0.0108 while learning that arm 3 is best. section 3.2
Figure 3.11 The ad selector testing multiple click models—or “arms”—using epsilon-greedy. With probability 1-epsilon, the system chooses the model with the highest mean CTR so far. Otherwise (probability epsilon) the system chooses with equal probability from the available models.
Visualizes an ad selector choosing the highest-mean-CTR arm most of the time and randomly exploring all available arms otherwise. section 3.2
Thompson sampling uses the same arm abstraction but selects through randomized probability matching: arms are chosen in proportion to the estimated probability that each is best. This can stop experimentation earlier while maintaining similar or slightly better CTR than epsilon-greedy. section 3.3.3 section 3.4
