A critic is a value network that predicts the expected return from a state. In actor-critic learning, it provides a baseline so the actor can judge whether an action did better or worse than expected. section 9.4
The critic and actor have distinct learning roles:
- Predict state values by minimizing squared error between predicted values and observed returns-to-go. section 9.4
- Provide the baseline for the advantage: subtract the predicted state value from the observed return. This relative signal reduces gradient variance without introducing bias. section 9.4
A better-than-expected action
If the critic predicts a return of −30 and the observed return is −15, the advantage is +15. The action performed better than expected, so the actor’s update increases its probability. section 9.4
Figure 9.5 Variance Reduction with Baselines.
A figure on how subtracting a baseline reduces variance in policy-gradient learning. section 9.4
