Contextual bandit

As explained in Machine Learning System Design

A sequential decision method that selects an action using the observed context and learns its outcome or reward.

A contextual bandit chooses among available actions using observed context, then uses the resulting feedback to pursue a reward. In recommendation systems, an agent consumes a user’s recent history and selects an ad banner, while optimizing measures such as click-through rate, conversion rate, or eCPM. Because shown ads influence later behavior, the decision process has a feedback loop. section 12.1.3

A practical contextual-bandit setup includes:

• User context, such as recent history. section 12.1.3
• Available items or ads, treated as alternative actions. section 12.1.3
• Observed feedback, which may be binary or continuous. section 12.1.3
• Historical replay for simulating recommendations and evaluating policies offline. section 12.1.3

Figure 12.1 Unbiased estimator (offline validation): stream of real events

Use the event-stream figure to contrast real recommendation events with their offline validation setting. section 12.1.3

How Machine Learning System Design covers it

How deep it goes

Explains the agent, context, actions, feedback loop, and replay simulation, but does not derive bandit algorithms.

Examples

Uses ad-banner selection with user history and CTR, CVR, or eCPM as reward measures.

What it assumes

Assumes familiarity with user features, available recommendations, observed feedback, and agent-environment interaction.

What you can do after

Evaluate recommendation policies offline by constructing a historical replay simulation.

How it connects

Connects contextual bandits with recommender systems, multi-armed bandits, reinforcement learning, and offline evaluation.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Experimentation for EngineersA sequential decision method that selects among alternatives using both observed outcomes and the context associated with each decision.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA contextual multi-armed bandit is a bandit model in which an agent observes contextual information before choosing an action and receiving action-specific feedback.

Compare how 3 books explain contextual bandit →

Related concepts

• multi-armed banditcontrasts with · A multi-armed bandit is a sequential decision problem in which an agent chooses among options with uncertain rewards, balancing reward seeking with gathering information about those options.
• study designmeasured by · Study design is the planned arrangement of procedures, conditions, measurements, and comparisons used to investigate a question and support valid conclusions.
• Vowpal Wabbitused by · Vowpal Wabbit is an open-source machine-learning system supporting scalable online learning and contextual bandit algorithms.
• complexityproduces · Complexity is the degree of difficulty in understanding, analyzing, changing, or operating a system or problem, arising from its structure, behavior, or interacting parts.
• reward signaluses · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.