liveBook · Manning

Constrained bandit

As explained in Applied Reinforcement Learning

A constrained bandit is a bandit model in which action selection must satisfy resource, exposure, fairness, or other operational constraints.

A constrained bandit extends ordinary bandit decision-making with operational limits. It chooses actions while respecting budgets, exposure caps, fairness goals, or similar constraints, rather than optimizing a single metric without safeguards. section 5.6

Use constrained bandits when business objectives extend beyond immediate clicks:

  • Shape rewards to reflect profit, customer lifetime value, margin protection, brand safety, or fairness instead of optimizing clicks alone. section 5.6
  • Apply explicit constraints when actions must stay within budgets, exposure limits, or fairness requirements. section 5.6
  • Recognize that delayed outcomes may require proxy rewards or buffered updates before the system can learn fairly. section 5.6

Marketing decisions

A marketing system might otherwise maximize clicks by giving away too many discounts or promoting unsuitable products. A constrained bandit instead balances response with profit, margin, brand safety, fairness, and limits on exposure or spending. section 5.6

How Applied Reinforcement Learning covers it

How deep it goes
Explains constraints and reward shaping through business examples, but stops short of algorithmic implementation or mathematical treatment.
Examples
Uses marketing clicks, discounts, product promotion, delayed purchases, budgets, exposure caps, and fairness goals.
What you can do after
Evaluate/understand when business objectives require constrained action selection rather than single-metric click optimization.
How it connects
Connects constrained bandits to reward shaping, proxy rewards, contextual bandits, combinatorial bandits, and adversarial bandits.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.