A marketing manager with several ads and no reliable click-through data must decide which ad to show each visitor. A multi-armed bandit models this as a repeated choice among actions, or arms: choose an ad, observe its reward, and update estimates of how well it performs. section 5.3 section 5.0
A pure multi-armed bandit looks for the best arm overall. It does not model state transitions or how a current action changes a later state. It fits repeated decisions with immediate payoffs, such as selecting ads, emails, layouts, or discounts. section 5.0 section 5.7
The central challenge is exploration versus exploitation. Trying less-tested arms can reveal better choices, while choosing the arm with the best current estimate earns rewards from what is already known. Committing too early can leave a better arm undiscovered. section 5.3 section 5.7
Common selection strategies balance this trade-off differently:
- Epsilon-greedy usually selects the best estimated arm but sometimes chooses randomly. section 5.3 section 5.7
- Upper confidence bound favors arms with high estimated rewards or high uncertainty. section 5.3 section 5.7
- Thompson sampling uses probability beliefs to balance exploration and exploitation. section 5.3 section 5.7
Five competing ads
Suppose five ads have unknown click-through rates. Each impression selects one ad; a click gives reward 1 and no click gives 0. Over 1,000 impressions, the agent estimates each rate from observed outcomes and adjusts its choices to increase total clicks. In the simulation, the hidden rates are 5%, 12%, 8%, 15%, and 11%, so the fourth ad is best, although the agent is not told this. section 5.3
Listing 5.1 The Ad Campaign Environment
class AdCampaignEnvironment:
def __init__(self, ad_probabilities): #A
self.ad_probabilities = ad_probabilities #A
self.num_ads = len(ad_probabilities) #A
def pull_arm(self, ad_index):
if ad_index < 0 or ad_index >= self.num_ads: #B
raise ValueError("Invalid ad index.") #B
ctr = self.ad_probabilities[ad_index] #C
if np.random.rand() < ctr: #D
return 1 #E
else:
return 0 #E
The simulated environment keeps each ad’s click probability hidden and returns a binary reward when an arm is chosen. section 5.3
A contextual bandit extends the decision by using features such as customer demographics or time of day to personalize the choice. In real campaigns, changing conditions and delayed rewards may also require adjustments, such as forgetting old observations or buffering feedback. section 5.7





