A bandit may have to choose before it can see the outcome: an email might be opened the next morning, or a purchase might follow several visits. This lag makes learning harder because feedback is not yet clear when the action is taken. section 5.6
Two practical ways to keep learning while outcomes are pending:
- Use proxy rewards, such as opens or add-to-cart events, as early signals. section 5.6
- Buffer pending outcomes until there is enough clarity to update fairly. section 5.6
