The Bellman equation turns long-horizon reward estimation into a recursive update: estimate an action by combining its immediate reward with the discounted value of the successor state. This makes value estimation operational for reinforcement learning and temporal-difference updates. section 10.4 section 6.2.1
For a policy π, the general action-value form is Qπ(st, at) = E[rt + γVπ(st+1) | st, at]. Q measures expected future reward for a state–action pair, while V measures expected future reward from a state. In deterministic transitions, the expectation simplifies to Q(st, at) = rt + γV(st+1). section 6.2.1
The equation supports several practical RL operations:
- Q-learning uses it to update estimates of how useful each state–action pair is as new experience arrives. section 10.4
- Temporal-difference learning replaces a complete return with the discrepancy between a value prediction and the observed reward plus successor value. section 10.4 section 6.2.1
- The same discrepancy defines the advantage estimate: A(st, at) = rt + γV(st+1) − V(st), measuring whether an action performs better than the state’s baseline. section 6.2.1
Interpreting a TD update
Suppose a value function predicts V(st) and an action produces reward rt before reaching st+1. The target rt + γV(st+1) is compared with V(st); their difference is the TD residual, which drives the value estimate toward what occurred. section 6.2.1
Equation 6.19
\[\label{eq:advantage_trick} A(s_t,a_t) = Q(s_t,a_t) - V(s_t) = r_t + \gamma V(s_{t+1}) - V(s_t).\]Inspect the advantage identity connecting Q, V, immediate reward, successor value, and the TD residual. section 6.2.1


