liveBook · Manning

Offline evaluation

As explained in Applied Reinforcement Learning

Offline evaluation estimates an agent's performance by replaying previously collected interaction data instead of interacting with a live environment.

Offline evaluation tests a bandit on historical interactions when a live experiment is unavailable. Replay each customer's context and let the agent choose a discount. Count the historical reward only when its choice matches the recorded discount; otherwise, record zero because the alternative reward is unknown. section 5.5

A practical evaluation separates learning from testing:

  • Train LinUCB on the first 80% of the data by updating it with historical context-action-reward tuples. section 5.5
  • Evaluate on the remaining 20%, comparing the pretrained contextual agent with a cold-start epsilon-greedy baseline. section 5.5

Personalized discounts

For each customer, an agent selects a discount. If it selects the discount given historically, the recorded reward contributes to its measured performance; a different choice receives zero. This conservative rule cannot reveal what the customer would have done under another discount. section 5.5

Figure 5.11 Performance comparison on the test set.

Compare the agents' cumulative rewards on the held-out test data. section 5.5

How Applied Reinforcement Learning covers it

How deep it goes
Details replay, reward handling, and an 80/20 train-test split, but cannot observe rewards for unchosen actions.
Examples
Uses customer discount choices, historical rewards, and cumulative test rewards to illustrate evaluation.
What it assumes
Assumes historical context-action-reward records and teaches evaluation by replaying those records.
What you can do after
Evaluate a contextual bandit against a baseline using a held-out historical dataset.
How it connects
Connects contextual bandits with LinUCB and epsilon-greedy comparison.

Also explained in

Compare how 5 books explain offline evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.