liveBook · Manning

Advantage function

As explained in Applied Reinforcement Learning

Advantage measures how much an action's expected return exceeds or falls below the value of its state under a policy.

A cooling controller needs to know whether a chosen fan speed is better than its usual choices for the current conditions. The advantage function makes that comparison: Aπ(s, a) = Qπ(s, a) − Vπ(s). Here, Qπ is the expected return for taking action a in state s and then following policy π; Vπ is the expected return from the state under that policy. section 4.2 section 9.3

A positive advantage means the action is better than the policy’s average for that state; a negative advantage means it is worse. The state value provides a baseline, so the comparison accounts for whether the situation itself was favorable or difficult. section 4.2 section 9.4

For a sampled trajectory, an estimate is Gₜ − V(sₜ), where Gₜ is the observed return-to-go. This compares one observed outcome with the expected state value; it need not equal the action’s expected advantage. A critic can learn the baseline by predicting returns, while an actor updates its policy using the advantage-weighted gradient. Subtracting the baseline reduces gradient variance without changing its expected value. section 9.4

Adjusting a fan-speed policy

Suppose a Gaussian policy has mean 0.7 and standard deviation 0.2, and samples fan speed 0.82. If its estimated advantage is +5, the policy-gradient update pushes the mean toward 0.82 and decreases the standard deviation. A negative advantage instead pushes the mean away from that sampled action. section 9.3

In a data-center cooling training loop, discounted returns are computed backward through each trajectory and the critic’s predictions are subtracted. Normalizing the resulting advantages to zero mean and unit standard deviation helps keep gradient magnitudes consistent across training. section 9.7

Listing 9.10 Policy gradient update with value baseline and entropy regularization
data = buffer.get_tensors()
T, B = returns.shape
obs_flat = data['observations'].view(
    T * B, config.n_zones, -1)                        #A
actions_flat = data['actions'].view(
    T * B, config.n_zones, 2)  #A
returns_flat = returns.view(T * B)
advantages_flat = advantages.view(T * B)
 
returns_mean = returns_flat.mean()
returns_std = returns_flat.std() + 1e-8
returns_normalized = (
    returns_flat - returns_mean) / returns_std  #B
 
log_probs, entropy, values = \
    model.evaluate_actions(obs_flat, actions_flat)  #C
 
policy_loss = -(
    log_probs * advantages_flat.detach()).mean()  #D
 
value_loss = F.mse_loss(
    values, returns_normalized.detach())  #E
 
entropy_loss = -entropy.mean()  #F
 
total_loss = (policy_loss +
    config.value_coef * value_loss +
    config.entropy_coef * entropy_loss)  #G
 
optimizer.zero_grad()
total_loss.backward()  #H
torch.nn.utils.clip_grad_norm_(
    model.parameters(), config.max_grad_norm) #I
optimizer.step()

The policy update weights action log-probabilities by the computed advantages, while a separate loss trains the value prediction. section 9.7

How Applied Reinforcement Learning covers it

How deep it goes
Defines expected advantage and develops a sampled baseline estimate through actor-critic updates and a cooling training loop.
Examples
Works through a Gaussian fan-speed update and a data-center cooling optimization loop.
What you can do after
Apply advantage estimates to weight policy updates and stabilize training with a learned value baseline.
How it connects
Connects advantage to state-value and action-value functions, policy gradients, REINFORCE, and actor-critic methods.

Also explained in

Compare how 5 books explain advantage function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.