A discount factor controls how much future rewards matter when an agent chooses an action. Think of choosing between one marshmallow now and more marshmallows later: a patient agent is willing to wait, while an impulsive one focuses on what it gets immediately. section 10.6
The setting changes how far ahead the agent values rewards:
- At 0, only the immediate reward matters. section 10.6
- At 0.9, future rewards still matter considerably; 0.9 is a typical default. section 10.11
- At 1, future rewards are not discounted in the delayed-gratification analogy. section 10.6
Updating the value of an eastward move
Suppose moving east gives an immediate reward of +1. In the new state, the highest predicted Q-value is 2.0. With a discount factor of 0.9, the training target is 1 + 0.9 × 2.0 = 2.8. If the network predicted 2.2 for the original move, training uses the squared difference between 2.2 and 2.8 as its loss. The factor lets the move receive credit for both its immediate reward and its promising next state. section 10.11
Future value decays with distance in time. In Q-learning, this encourages the agent to reach a rewarding goal by a shorter route rather than wander before collecting the reward. section 10.6
