liveBook · Manning

Delayed or censored reward

As explained in Applied Reinforcement Learning

A delayed reward is feedback whose value becomes observable only after a time interval following the associated action.

A bandit may have to choose before it can see the outcome: an email might be opened the next morning, or a purchase might follow several visits. This lag makes learning harder because feedback is not yet clear when the action is taken. section 5.6

Two practical ways to keep learning while outcomes are pending:

  • Use proxy rewards, such as opens or add-to-cart events, as early signals. section 5.6
  • Buffer pending outcomes until there is enough clarity to update fairly. section 5.6

How Applied Reinforcement Learning covers it

How deep it goes
Explains why delayed feedback complicates learning and names proxies and buffers, but does not detail their implementation.
Examples
Uses next-morning email opens, purchases after multiple visits, and opens or add-to-cart events as early signals.
What you can do after
Apply proxy rewards or buffer pending outcomes to keep bandit learning timely and fair.
How it connects
Connects delayed feedback to bandits, proxy rewards, and learning updates.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.