A discount factor, commonly written as gamma, multiplies rewards so temporally distant rewards contribute less to learning or evaluation. In Q-learning, gamma controls how strongly future rewards influence the value of an action alongside the immediate reward. section 11.1.1
The same idea serves two related purposes:
- Q-learning uses gamma in its update rule: the value estimate combines the observed reward with gamma times the best value of the next state. section 11.1.1
- A NEAT lunar-lander experiment uses a genome-specific discount to calculate normalized discounted rewards for fitness evaluation rather than directly selecting actions. section 11.4
Frozen Lake
A Q-learning agent sets gamma to 0.95, trains for 20,000 episodes, and applies the factor while updating a state-action Q-table. The resulting values support later greedy action selection. section 11.1.1
Listing 11.4 EDL_11_1_FrozenLake.ipynb: Defining hyperparameters
total_episodes = 20000 #1 learning_rate = 0.7 #2 max_steps = 99 #3 gamma = 0.95 #4 epsilon = 1.0 #5 max_epsilon = 1.0 #5 min_epsilon = 0.01 #5 decay_rate = 0.005 #5
Inspect the hyperparameter listing to see gamma alongside the learning rate, episode count, step limit, and exploration settings. section 11.1.1
