An action-value function, or Q-function, maps a state–action pair to a scalar estimate of future reward. It answers how useful taking action a in state s is, under policy π. Unlike a value function, which evaluates only states, Q evaluates the immediate choice and its downstream consequences. section 10.4 section 6.2.1
The Bellman relationship makes Q operational: Qπ(st, at) = E[rt + γVπ(st+1) | st, at]. The estimate combines the reward produced now with the discounted value of the next state. With deterministic transitions, the expectation simplifies to rt + γV(st+1). For a deterministic policy, Vπ(st) = Qπ(st, π(st)); for a stochastic policy, Vπ averages Q over actions sampled from π. section 6.2.1
Q-values support several reinforcement-learning operations:
- Q-learning updates estimates of state–action usefulness through temporal-difference learning, allowing policy updates as new data arrives rather than relying only on a fixed cumulative dataset. section 10.4
- Policy-gradient methods can use Qπ(st, at) as a per-timestep learning signal, trading against alternatives such as total return, baselines, advantage, or a TD residual. section 6.2.1
- The advantage function centers Q by subtracting the state value: A(st, at) = Q(st, at) − V(st). This measures whether an action performs better or worse than the state’s average expectation. section 6.2.1
From Q-value to policy update
Suppose a policy takes action at in state st. A learned value model estimates V(st) and V(st+1), while the environment supplies reward rt. The TD residual rt + γV(st+1) − V(st) estimates the action’s advantage and can weight the action’s log-probability contribution during policy optimization. section 6.2.1
Equation 6.19
\[\label{eq:advantage_trick} A(s_t,a_t) = Q(s_t,a_t) - V(s_t) = r_t + \gamma V(s_{t+1}) - V(s_t).\]Use the equation to inspect the equivalence between the Q-minus-V advantage and the one-step TD residual. section 6.2.1
