A discount factor, denoted γ ∈ [0,1], is a multiplier that determines how much a reward received later contributes relative to reward received now. In a sequential reinforcement-learning objective, return is the weighted sum of rewards along a trajectory: rewards k steps ahead receive weight γ^k. This makes γ a direct modeling choice about the balance between near-term and long-term behavior. section 3.1 section 6.2
Equation 6.2
\[\label{eq:return_definition} G_t = R_{t+1} + \gamma R_{t+2} + \cdots = \sum_{k=0}^\infty \gamma^k R_{t+k+1}.\]The return equation makes the weighting explicit: each additional step into the future adds another factor of γ. section 6.2
The boundary values and common setting have distinct operational meanings:
- γ = 0: only the immediate reward matters. section 6.2
- γ = 1: future rewards are weighted equally with immediate rewards. section 6.2
- γ close to 1, such as 0.99: emphasizes long-term stable behavior while retaining short-horizon learning feedback. section 6.2
- γ < 1: keeps an infinite-horizon continuing-task objective well-defined. section 3.1
Discounting also gives return a useful recursive form: the return at time t is the current reward plus γ times the next return. Value functions use this discounted future return as their prediction target, and policy-gradient methods optimize policies for expected discounted return. In token-level PPO, a backward pass can compute per-token Monte Carlo targets by adding each reward to γ times the next nonterminal target; advantages are then return minus the value prediction. section 6.2 section 6.2.7
Why γ is usually 1 for single-turn RLHF
Standard RLHF commonly treats one prompt and its entire completion as a single action with one response-level reward. Setting γ = 1 lets every generated token belong to that same action. Discounting earlier tokens would arbitrarily reduce their contribution when the reward evaluates the collective completion. In genuinely multistep agentic settings involving tool calls, code execution, or web browsing, distinct actions can again make discounting relevant. section 3.1.3 section 6.3.2
Do not treat γ as universally beneficial: it encodes a sequential-decision assumption. It is essential for balancing short- and long-term rewards in multistep RL, but can be inappropriate when a score belongs to a whole sequence rather than individual decisions. Loss aggregation and whether KL penalties are folded into rewards are separate implementation choices from the discount factor itself. section 6.3.2 section 10.3
