Return is the reward aggregate used to judge an interaction sequence. At time t, the discounted return G_t sums each future reward with successively larger powers of γ: G_t = Σₖ γᵏRₜ₊ₖ₊₁. γ controls the horizon: zero values immediate reward only, while one weights future rewards equally. section 6.2 section 3.1.2
Return connects directly to reinforcement-learning objectives and estimators:
- Value functions estimate expected return from a state: V(s) = E[G_t | S_t = s]. section 6.2
- Policy-gradient methods maximize expected trajectory return and use sampled rollouts to estimate that objective. section 6.2 section 6.2.1
- Advantages compare a return target with a value prediction, producing the policy-learning signal A_t = G_t − V(s_t). section 6.2.7
CartPole and token-level returns
In CartPole, early random episodes end quickly, whereas corrective pushes keep the pole balanced longer and increase cumulative reward. In PPO, a reverse-time loop accumulates post-KL per-token rewards, resets at terminals, applies γ, and stores each running value as a return target. section 3.1.2 section 6.2.7
Equation 6.2
\[\label{eq:return_definition} G_t = R_{t+1} + \gamma R_{t+2} + \cdots = \sum_{k=0}^\infty \gamma^k R_{t+k+1}.\]Use the formal return equation to inspect discounted reward accumulation across future timesteps. section 6.2
