Discount factor
A discount factor is a numerical multiplier that weights future rewards or utility relative to present rewards or utility, typically reducing the contribution of more distant outcomes.
Grokking AI Algorithms, Second Edition
A discount factor is a reinforcement-learning hyperparameter that determines how strongly future rewards contribute to the value of a current decision.
- How it is used
- Controls the weight of future rewards when training action values with Q-learning or an ANN.
- How deep it goes
- Defines settings intuitively, notes exponential decay, and works through a one-step training target without a theoretical derivation.
- Examples
- Uses a marshmallow choice and an eastward move with a +1 reward, 2.0 next-state maximum, and 0.9 discount.
- What you can do after
- Apply it to weight future rewards in Q-learning and calculate a one-step ANN training target.
Read how Grokking AI Algorithms, Second Edition explains discount factor →
Optimization Algorithms: AI techniques for design, planning, and control problems
A discount factor is a scalar between zero and one that determines how strongly future rewards contribute to an action value.
- How it is used
- Weights future rewards in value estimates and in a CartPole A2C advantage calculation.
- How deep it goes
- Defines the near-zero versus near-one trade-off and shows gamma in code, without deriving a tuning rule.
- Examples
- Uses a CartPole A2C implementation with gamma set to 0.99 and applied to the next-state critic estimate.
- What you can do after
- Build it from scratch into the shown CartPole A2C advantage calculation by setting gamma and weighting the next-state value.
Evolutionary Deep Learning: Genetic algorithms and neural networks
A discount factor, commonly denoted gamma, is a numerical multiplier that reduces the contribution of temporally distant rewards.
- How it is used
- Controls the influence of future rewards in Q-learning and evaluates the importance of past actions in an evolutionary fitness calculation.
- How deep it goes
- Defines the mechanism and shows executable update and fitness formulas, while deferring theoretical derivation and broader limitations.
- Examples
- Uses a Frozen Lake Q-table with gamma 0.95 and a Lunar Lander genome whose discount is initialized, crossed over, mutated, and bounded.
- What you can do after
- Apply it by configuring Q-learning updates or by computing discounted rewards for comparing evolutionary-agent fitness.
Reinforcement Learning from Human Feedback
A discount factor is a scalar between zero and one that weights future rewards relative to nearer rewards.
- How it is used
- Weights future rewards when defining returns and objectives for sequential reinforcement learning.
- How deep it goes
- Derives discounted and recursive returns, then implements backward PPO critic targets; defers generalized advantage estimation details.
- Examples
- Contrasts multistep trajectories with single prompt-completion RLHF and shows a backward per-token return computation.
- What you can do after
- Operate/configure it by choosing γ for multistep returns or setting γ = 1 for response-level RLHF.
Read how Reinforcement Learning from Human Feedback explains discount factor →
Related concepts
- discounted rewarduses · A discounted reward is a reward whose contribution is multiplied by a temporal discount factor.
- hyperparameterkind of · A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.
- value functionused by · A value function assigns or estimates the expected cumulative reward, utility, or quality associated with a state or action under a policy or decision rule.
- target q-valueconfigures · A target Q-value is the reinforcement-learning estimate formed from an immediate reward plus a discounted estimate of the best future action value.
- Bellman equationconfigures · A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
- reward signalconfigures · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.