liveBook · Manning

Beta distribution

As explained in Applied Reinforcement Learning

A continuous probability distribution on the interval from zero to one, commonly used to represent uncertainty about a Bernoulli probability.

A Beta distribution represents uncertainty about an unknown probability, such as an ad’s click-through rate. It stays between zero and one and updates as clicks and non-clicks are observed. section 5.3

In Thompson sampling, each ad has a Beta belief. The algorithm samples a possible click-through rate for each ad, chooses the ad with the highest sample, then updates its belief using the result. section 5.3

Listing 5.6 The Thompson Sampling Agent
class ThompsonSamplingAgent:
    def __init__(self, num_ads):
        self.num_ads = num_ads
        self.alphas = [1] * num_ads #A
        self.betas = [1] * num_ads #A
 
    def choose_action(self):
        sampled_ctrs = [np.random.beta(self.alphas[i],
        [CA]self.betas[i]) for i in range(self.num_ads)] #B
        return np.argmax(sampled_ctrs) #C
 
    def update(self, chosen_ad_index, reward):
        if reward == 1:
            self.alphas[chosen_ad_index] += 1 #D
        else:
            self.betas[chosen_ad_index] += 1 #E

See the agent code for Beta sampling and updates after clicks or non-clicks. section 5.3

How Applied Reinforcement Learning covers it

How deep it goes
Defines a probability belief and demonstrates sampling and success/failure updates in code.
Examples
Uses ad click-through rates, with clicks and non-clicks as the observed outcomes.
What it assumes
Assumes binary click outcomes and explains the Bernoulli setup and unknown success probability.
What you can do after
Apply it via Thompson sampling to sample ad click-through rates, choose an ad, and update beliefs from outcomes.
How it connects
Connects Beta beliefs to Bernoulli outcomes and Thompson sampling.

Also explained in

Compare how 4 books explain beta distribution →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.