liveBook · Manning

Reward-to-go

As explained in Reinforcement Learning from Human Feedback

Reward-to-go is the cumulative future reward expected from a specified point in a trial or trajectory.

Reward-to-go is the cumulative future reward expected from a specified point in a trial or trajectory. It combines immediate reward with multistep optimization, giving reinforcement-learning methods a target for estimating how much future benefit remains. section 10.3

In reinforcement learning, reward-to-go connects several standard components:

  • Reward functions assign scalar evaluations to actions, while reward-to-go aggregates expected rewards over a future time window. section 10.3
  • A discount factor γ ∈ [0,1] multiplicatively reweights future rewards, controlling how strongly later outcomes contribute. section 10.3
  • Bellman equations recursively estimate reward-to-go in Markov decision processes, while temporal-difference learning updates estimates as new data arrives. section 10.4 section 10.4

State–action estimates

Q-learning estimates the reward-to-go associated with each state–action pair through Bellman-based temporal-difference updates; its Q-function maps a state and action to a scalar future-reward estimate. section 10.4

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines cumulative future reward, introduces discounting, and connects estimation to Bellman recursion and temporal-difference updates.
Examples
Uses Q-learning, state–action value estimates, and Backgammon as concrete anchors for future-reward estimation.
What you can do after
Evaluate how RL methods estimate and update future reward under Markov decision process formulations.
How it connects
Connects reward-to-go with reward functions, utility, discount factors, Markov decision processes, value functions, and temporal-difference learning.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.