liveBook · Manning

Bandit feedback

As explained in Experimentation for Engineers

Bandit feedback is observational feedback containing rewards for policy-selected actions while omitting outcomes for unsampled actions.

Bandit feedback records the reward for the action a policy selected while omitting outcomes for actions it did not sample. In a recommender, the logged data therefore contains context, chosen action, and reward—not complete labels for every possible recommendation. section 5.2.2

This creates three practical consequences:

  • Treat the data as policy-collected rather than randomly sampled, so model estimates can be biased toward previous decisions. section 5.2.2
  • Use exploration to select actions the current model would not choose and collect their previously missing counterfactual rewards. section 5.2.2
  • Refit on the expanding logs so broader action and context coverage can reduce bias and improve future rewards. section 5.2.2

Recommender exploration

An epsilon-greedy recommender randomly selects an action with probability epsilon and otherwise follows its prediction-based choice. The random selections expose posts that a greedy policy would omit, adding counterfactual observations to later training logs. section 5.2.2

How Experimentation for Engineers covers it

How deep it goes
Defines missing counterfactual outcomes and traces how exploration, refitting, and broader coverage reduce bias; it stops short of formal estimators.
Examples
Uses a greedy recommender, an epsilon-greedy recommender, recommendation logs, and mean viewing time.
What it assumes
Assumes familiarity with features, actions, rewards, policies, and supervised learning; distinguishes their roles in contextual-bandit data.
What you can do after
Apply epsilon-greedy exploration to gather counterfactual logs and improve a biased recommender policy.
How it connects
Connects bandit feedback to contextual bandits, policy sampling, supervised learning, counterfactuals, exploration, and reward modeling.
Cover of Experimentation for Engineers
liveBookmanning.com
Read Experimentation for Engineers on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Experimentation for Engineers. The section links point to the parts of the book it draws on.