The discount factor, γ, controls how much future rewards count when an agent evaluates a state or action. It is typically between 0 and 1. A value close to 0 favors immediate rewards; a value close to 1 gives more weight to rewards that arrive later. section 12.1.2
Discounting is part of expected cumulative reward. A state-value function V(s) estimates that reward from a state while following a policy; an action-value function Q(s, a) estimates it after taking a particular action. Changing γ therefore changes how strongly those estimates value future rewards. section 12.1.2
CartPole advantage calculation
In a CartPole A2C implementation, γ is set to 0.99. The advantage calculation combines the current reward with γ times the critic’s estimate of the next state, then subtracts the critic’s estimate of the current state. The next-state term is omitted when the episode is done. This is where the preference for future reward enters the training update. section 12.3
advantage = reward + (1-
done)*gamma*critic(torch.from_numpy(next_state).float()) -
critic(torch.from_numpy(state).float()) #1
total_reward += reward #2
state = next_state #3
#1 Compute the advantage function.
#2 Update the total reward.
#3 Move to the next state.
Inspect the advantage expression to see where gamma scales the next-state value and how the done flag removes that term. section 12.3
