liveBook · Manning

Multi-armed bandit

As explained in Applied Reinforcement Learning

A multi-armed bandit is a sequential decision model in which an agent repeatedly chooses among actions and observes their rewards without modeled state transitions.

A marketing manager with several ads and no reliable click-through data must decide which ad to show each visitor. A multi-armed bandit models this as a repeated choice among actions, or arms: choose an ad, observe its reward, and update estimates of how well it performs. section 5.3 section 5.0

A pure multi-armed bandit looks for the best arm overall. It does not model state transitions or how a current action changes a later state. It fits repeated decisions with immediate payoffs, such as selecting ads, emails, layouts, or discounts. section 5.0 section 5.7

The central challenge is exploration versus exploitation. Trying less-tested arms can reveal better choices, while choosing the arm with the best current estimate earns rewards from what is already known. Committing too early can leave a better arm undiscovered. section 5.3 section 5.7

Common selection strategies balance this trade-off differently:

Five competing ads

Suppose five ads have unknown click-through rates. Each impression selects one ad; a click gives reward 1 and no click gives 0. Over 1,000 impressions, the agent estimates each rate from observed outcomes and adjusts its choices to increase total clicks. In the simulation, the hidden rates are 5%, 12%, 8%, 15%, and 11%, so the fourth ad is best, although the agent is not told this. section 5.3

Listing 5.1 The Ad Campaign Environment
class AdCampaignEnvironment:
    def __init__(self, ad_probabilities): #A
        self.ad_probabilities = ad_probabilities #A
        self.num_ads = len(ad_probabilities) #A
 
    def pull_arm(self, ad_index):
        if ad_index < 0 or ad_index >= self.num_ads: #B
            raise ValueError("Invalid ad index.") #B
 
 
        ctr = self.ad_probabilities[ad_index] #C
 
        if np.random.rand() < ctr: #D
            return 1 #E
        else:
            return 0 #E

The simulated environment keeps each ad’s click probability hidden and returns a binary reward when an arm is chosen. section 5.3

A contextual bandit extends the decision by using features such as customer demographics or time of day to personalize the choice. In real campaigns, changing conditions and delayed rewards may also require adjustments, such as forgetting old observations or buffering feedback. section 5.7

How Applied Reinforcement Learning covers it

How deep it goes
Defines the decision cycle and trade-off, then implements and compares agents in an ad campaign simulation.
Examples
Uses five banner ads with binary click rewards and hidden click-through rates over 1,000 impressions.
What you can do after
Builds a bandit simulation and selection agents from scratch to experiment with ad selection.
How it connects
Connects exploration–exploitation to contextual bandits, epsilon-greedy, upper confidence bounds, and Thompson sampling.

Also explained in

Compare how 6 books explain multi-armed bandit →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.