liveBook · Manning

Bandit algorithm

As explained in Applied Reinforcement Learning

A bandit algorithm selects among actions and updates its action-selection strategy from observed rewards while balancing exploration and exploitation.

A bandit algorithm learns while choosing: it selects an action, observes its immediate reward, and updates its strategy. Its central challenge is balancing exploration of uncertain options with exploitation of options that have performed well. section 5.1 section 5.7

Common decision strategies include:

  • Epsilon-greedy explores occasionally while usually choosing the current favorite. section 5.7
  • UCB directs exploration toward options with greater uncertainty. section 5.7
  • Thompson Sampling uses a Bayesian approach to balance exploration and exploitation. section 5.7

Business decisions

Choosing among ads, emails, layouts, or discounts can use immediate rewards to improve decisions over time. Contextual bandits can use information such as demographics or time of day to personalize choices. section 5.7

Figure 5.1 Comparing multi-armed bandits, contextual bandits, and full reinforcement learning.

Compare multi-armed bandits, contextual bandits, and full reinforcement learning by the role of state and sequential planning. section 5.1

How Applied Reinforcement Learning covers it

How deep it goes
Explains the choose-observe-update loop, algorithm families, incremental updates, and practical evaluation limits.
Examples
Uses ads, emails, layouts, and discounts as immediate-payoff decisions, including contextual discount targeting with LinUCB.
What it assumes
Assumes immediate rewards and static or absent state; contrasts these with dynamic states and multi-step planning.
What you can do after
Apply bandit strategies to immediate-payoff decisions and assess real campaigns with business KPI proxies.
How it connects
Connects bandit algorithms to reinforcement learning, contextual and multi-armed bandits, exploration-exploitation, regret, and off-policy evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.