liveBook · Manning

Reward-to-go

As explained in Applied Reinforcement Learning

Return-to-go is the discounted sum of rewards from a specified decision time onward in a trajectory.

Reward-to-go, written Gₜ, is the discounted sum of rewards from time t onward. In policy-gradient estimates, it credits an action only for later rewards, avoiding noise from rewards received before that action. section 9.3

Gₜ = rₜ + γ rₜ₊₁ + γ² rₜ₊₂ + ... = Σₖ₌ₜᵀ γᵏ⁻ₜ rₖ

The equation expresses reward-to-go as the sum of future rewards, discounted by their distance from time t. section 9.3

How Applied Reinforcement Learning covers it

How deep it goes
Defines the discounted sum and explains its credit-assignment rationale, while leaving detailed calculations to a separate walkthrough.
What you can do after
Understand why gradient estimates use future rewards rather than the full trajectory return.
How it connects
Connects Monte Carlo estimation, policy gradients, REINFORCE, and variance reduction.

Also explained in

Compare how 2 books explain reward-to-go →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.