A target Q-value estimates what an action should be worth: its immediate reward plus a discounted estimate of the best future action value. It gives a neural network a learning target instead of a fixed label. section 10.11
The calculation follows a simple sequence:
- Take the reward received after the action. section 10.11
- Find the largest predicted Q-value for the resulting state. section 10.11
- Discount that future value, then add it to the immediate reward. section 10.11
- Compare the target with the network’s prediction for the action taken; their squared difference becomes the loss. section 10.11
A move toward the east
Suppose an agent moves east, receives +1, and reaches a state whose predicted action values are North 1.2, South 0.5, East 2.0, and West 0.3. With a discount factor of 0.9, the target is 1 + 0.9 × 2.0 = 2.8. If the network predicted 2.2 for the original east action, the difference drives learning. section 10.11
