A contextual bandit chooses an action after observing context, then receives feedback for the chosen action. Its goal is to earn more reward over repeated choices. Unlike a noncontextual multi-armed bandit, an action’s expected reward can depend on the current context, so the best choice can change. Unlike a Markov decision process, the choice does not require planning through future state transitions. section 12.1 section 12.1.6
Choose a delivery truck
Suppose the route is city or interstate. For each delivery, choose a truck size, engine, and tire type. With three sizes, three engines, and four tire types, there are 36 possible combinations. The reward represents the suitability of the selected combination, with added noise for uncertain conditions. The intended decision is the best combination for the observed route—not one fixed truck for every route. section 12.5
A practical selection loop is:
- Generate an example for the shared route context and each available truck combination. section 12.5
- Predict a probability distribution over combinations and sample an action from it. Sampling leaves room to explore rather than always taking the current favorite. section 12.5
- Observe the chosen truck’s reward and use that feedback to continue training. Repeat, then test a choice for a specified route. section 12.1.6 section 12.5
def generate_combinations(shared_context, size_types, engine_types, tire_types):
examples = [f"shared |User {shared_context}"]
descriptions = []
for i, size in enumerate(size_types):
for j, engine in enumerate(engine_types):
for k, tire in enumerate(tire_types):
examples.append(f"|Action truck_size={size} engine={engine}
tire={tire}")
descriptions.append((i, j, k))
return examples, descriptions
The action-generation listing shows how one shared context is paired with every size, engine, and tire combination before prediction. section 12.5
The Python implementation uses Vowpal Wabbit’s contextual-bandit exploration mode, an epsilon setting of 0.2, and PyTorch to sample from predicted probabilities. Try changing the available options or exploration setting and observing the resulting rewards. Check the reward signal before interpreting route-specific results: the displayed toy reward function accepts the route context but calculates reward from truck-option values without using that context. As written, it cannot demonstrate that different routes favor different trucks. section 12.5
