Temporal-difference (TD) error measures the mismatch between a value prediction and a one-step target: δₜ = rₜ + γV(sₜ₊₁) − V(sₜ). The target combines the immediate reward with a bootstrapped estimate of the next state, so the residual indicates whether the current value estimate was too low or too high. section 6.2.1 section 6.4.1
The residual connects value estimation to policy optimization in several ways:
- It provides a one-step advantage estimate: A(sₜ,aₜ) = Q(sₜ,aₜ) − V(sₜ) = rₜ + γV(sₜ₊₁) − V(sₜ). section 6.2.1
- It trades bias against variance: single-step TD has lower variance but greater bias than estimates using longer returns. section 6.4.1
- Generalized advantage estimation combines future TD errors with exponentially decaying weights, producing Âₜ = Σₗ₌₀∞(γλ)ˡδₜ₊ₗᵛ. section 6.4.1
Token-level GAE update
For each token position, compute delta = reward + gamma × not_done × next_value − value, then accumulate gae = delta + gamma × lambda × not_done × gae while iterating backward. Terminal masks stop bootstrapping and reset accumulation; the resulting advantages can become value-regression targets after adding the current values. section 6.4.1
Equation 6.55
\[\label{eq:TD_RESIDUAL} \delta_t^V = r_t + \gamma V(s_{t+1}) - V(s_t).\]Inspect the formal one-step TD-residual equation and its placement within multistep advantage estimation. section 6.4.1
