Contextual bandit

As explained in Experimentation for Engineers

A sequential decision method that selects among alternatives using both observed outcomes and the context associated with each decision.

A contextual bandit makes targeted, repeated decisions by predicting a short-term business-metric outcome for each available action in the current context. It fits prediction models from logged decisions and rewards offline, then uses a policy online to choose actions. This suits personalized services such as content, advertising, music, and movie recommendation, where different users may prefer different options. section 5.0 section 5.1

Figure 5.1 A generic contextual bandit (CB). A CB consists of offline and online parts. In the offline part, you fit a model that predicts the business metric. Online, the system usually makes prediction-based decisions that exploit the model to increase the business metric. Sometimes it makes decisions randomly—it “explores”—to collect novel data for the next offline fit.

The system alternates between fitting a reward-prediction model from logs and making online decisions; occasional random actions collect new data. section 5.1

In a recommender, the user is the context, a displayed post is the action, and viewing time is the reward. A model predicts reward from the context and action; a greedy policy selects the action with the largest predicted reward. Models can be refit daily as fresh logs arrive. section 5.1.1 section 5.1.2

Greedy social-media recommender

Represent each user with a feature vector and log each displayed post with its viewing time. Fit a separate reward model for each post from its logged contexts and rewards. For an arriving user, score every post and display the one with the highest predicted viewing time. Random initial model parameters can produce initial logs, but performance begins poorly before the system learns from them. section 5.1.1 section 5.1.2 section 5.1.3

Pure exploitation creates missing counterfactuals: the system observes rewards only for what it chose to show. Epsilon-greedy exploration addresses this by choosing a random action a small fraction of the time and using the greedy policy otherwise. The resulting data can reduce prediction bias and feedback loops, though exploration trades some immediate reward for information. section 5.0 section 5.2.2

Use a contextual bandit under these operating constraints:

• Optimize a metric directly caused by one action, such as an ad click or starting a suggested song. section 5.1.3
• Do not treat it as a solution for multistep returns where earlier actions change the meaning or consequences of later ones. section 5.1.3
• Plan for cold start, model fitting, exploration, and policy constraints such as safety or business rules. section 5.1.2 section 5.4
• Validate any system change with an A/B test, including new features, exploration changes, or safety checks. section 5.4

A contextual bandit extends a multi-armed bandit by conditioning each action on context. It is also a special case of reinforcement learning in which decisions do not affect the next context, and it differs from supervised learning because its training samples are selected by a policy rather than randomly sampled. section 5.2.2

How Experimentation for Engineers covers it

How deep it goes

Works through greedy modeling, online action selection, simulation, epsilon-greedy exploration, and A/B validation; defers full multistep reinforcement learning.

Examples

Uses a social-media recommender that predicts viewing time from user features and displayed posts, simulated over 30 days.

What it assumes

Assumes supervised-learning skills including feature engineering, selection, regularization, model selection, and architecture design.

What you can do after

Build it from scratch as a greedy or epsilon-greedy recommender that logs rewards, refits models, and selects actions.

How it connects

Connects multi-armed bandits, reinforcement learning, supervised learning, epsilon-greedy exploration, Thompson sampling, and A/B testing.

liveBookmanning.com

Read Experimentation for Engineers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA contextual multi-armed bandit is a bandit model in which an agent observes contextual information before choosing an action and receiving action-specific feedback.
• 

  liveBookmanning.com
  Machine Learning System DesignA sequential decision method that selects an action using the observed context and learns its outcome or reward.

Compare how 3 books explain contextual bandit →

Related concepts

• multi-armed banditcontrasts with · A multi-armed bandit is a sequential decision problem in which an agent chooses among options with uncertain rewards, balancing reward seeking with gathering information about those options.
• study designmeasured by · Study design is the planned arrangement of procedures, conditions, measurements, and comparisons used to investigate a question and support valid conclusions.
• Vowpal Wabbitused by · Vowpal Wabbit is an open-source machine-learning system supporting scalable online learning and contextual bandit algorithms.
• complexityproduces · Complexity is the degree of difficulty in understanding, analyzing, changing, or operating a system or problem, arising from its structure, behavior, or interacting parts.
• reward signaluses · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.

This explainer was generated by Manning's knowledge graph from the text of Experimentation for Engineers. The section links point to the parts of the book it draws on.