liveBook · Manning

Sparse reward

As explained in Applied Reinforcement Learning

A sparse reward is a feedback signal provided at only a small number of points, rather than continuously throughout a task.

A sparse reward gives feedback at only a few points in a multi-step task, often only at the end. The agent must learn from outcomes rather than feedback at every decision. section 2.6.1 section 10.3

Reward timing is a design choice. Delivery fuel costs might be measured only after a route, while daily stock rewards or penalties provide intermediate guidance. When direct feedback is delayed, measures such as clicks or engagement can serve as approximations. section 2.6.1

Figure 10.2 Language generation mapped as a Markov Decision Process.

Language generation extends the state with each token, then provides one reward after the complete response. section 10.3

How Applied Reinforcement Learning covers it

How deep it goes
Defines sparse rewards through comparison with stepwise feedback and illustrates both, without detailing learning algorithms.
Examples
Uses delivery routes, retailer inventory, generated responses, and a chess analogy to illustrate delayed feedback.
What you can do after
Evaluate whether a task receives feedback only at completion or at intermediate steps.
How it connects
Connects sparse rewards to episodic tasks, dense rewards, reward shaping, and RLHF.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.