liveBook · Manning

Critic

As explained in Applied Reinforcement Learning

The value-estimating component of an actor-critic system that predicts expected returns and supplies a learning baseline.

A critic is a value network that predicts the expected return from a state. In actor-critic learning, it provides a baseline so the actor can judge whether an action did better or worse than expected. section 9.4

The critic and actor have distinct learning roles:

  • Predict state values by minimizing squared error between predicted values and observed returns-to-go. section 9.4
  • Provide the baseline for the advantage: subtract the predicted state value from the observed return. This relative signal reduces gradient variance without introducing bias. section 9.4

A better-than-expected action

If the critic predicts a return of −30 and the observed return is −15, the advantage is +15. The action performed better than expected, so the actor’s update increases its probability. section 9.4

Figure 9.5 Variance Reduction with Baselines.

A figure on how subtracting a baseline reduces variance in policy-gradient learning. section 9.4

How Applied Reinforcement Learning covers it

How deep it goes
Train the value network by squared-error regression on observed returns, without detailing its network internals.
Examples
Use a predicted return of −30 and an observed return of −15 to demonstrate a positive advantage of 15.
What you can do after
Understand how a learned baseline stabilizes actor learning by expressing returns as advantages.
How it connects
Connects actor-critic learning with policy gradients, state-value functions, baselines, and advantages.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.