Context is observed information describing the current situation and conditioning an action’s expected reward. In a contextual multi-armed bandit, the best action can change with the context. section 12.1
Distinguish context from related decision settings:
- Treat a noncontextual multi-armed bandit as having one repeated decision situation with no changing state or side information. section 12.1.6
- Use context when action quality depends on current conditions, such as selecting a truck configuration for a city or interstate route. section 12.1 section 12.5
- Keep the setting distinct from a Markov decision process, where actions affect the environment across multiple time steps through states and transitions. section 12.1
Truck selection
Represent city or interstate as shared context, then evaluate combinations of truck size, engine type, and tire type. A contextual bandit samples an action, observes a noisy reward, and updates its choices. A Vowpal Wabbit implementation uses action-dependent features, exploration, and 2,500 training iterations; one test selects a medium electric truck with snow tires for a city context. section 12.5
Figure 12.12 A CMAB-based recommender system to select delivery truck size, engine type, and tire type based on a specified context
Explore the contextual-bandit recommender diagram linking route context to truck size, engine, and tire decisions. section 12.5
