A cooling controller needs to know whether a chosen fan speed is better than its usual choices for the current conditions. The advantage function makes that comparison: Aπ(s, a) = Qπ(s, a) − Vπ(s). Here, Qπ is the expected return for taking action a in state s and then following policy π; Vπ is the expected return from the state under that policy. section 4.2 section 9.3
A positive advantage means the action is better than the policy’s average for that state; a negative advantage means it is worse. The state value provides a baseline, so the comparison accounts for whether the situation itself was favorable or difficult. section 4.2 section 9.4
For a sampled trajectory, an estimate is Gₜ − V(sₜ), where Gₜ is the observed return-to-go. This compares one observed outcome with the expected state value; it need not equal the action’s expected advantage. A critic can learn the baseline by predicting returns, while an actor updates its policy using the advantage-weighted gradient. Subtracting the baseline reduces gradient variance without changing its expected value. section 9.4
Adjusting a fan-speed policy
Suppose a Gaussian policy has mean 0.7 and standard deviation 0.2, and samples fan speed 0.82. If its estimated advantage is +5, the policy-gradient update pushes the mean toward 0.82 and decreases the standard deviation. A negative advantage instead pushes the mean away from that sampled action. section 9.3
In a data-center cooling training loop, discounted returns are computed backward through each trajectory and the critic’s predictions are subtracted. Normalizing the resulting advantages to zero mean and unit standard deviation helps keep gradient magnitudes consistent across training. section 9.7
Listing 9.10 Policy gradient update with value baseline and entropy regularization
data = buffer.get_tensors()
T, B = returns.shape
obs_flat = data['observations'].view(
T * B, config.n_zones, -1) #A
actions_flat = data['actions'].view(
T * B, config.n_zones, 2) #A
returns_flat = returns.view(T * B)
advantages_flat = advantages.view(T * B)
returns_mean = returns_flat.mean()
returns_std = returns_flat.std() + 1e-8
returns_normalized = (
returns_flat - returns_mean) / returns_std #B
log_probs, entropy, values = \
model.evaluate_actions(obs_flat, actions_flat) #C
policy_loss = -(
log_probs * advantages_flat.detach()).mean() #D
value_loss = F.mse_loss(
values, returns_normalized.detach()) #E
entropy_loss = -entropy.mean() #F
total_loss = (policy_loss +
config.value_coef * value_loss +
config.entropy_coef * entropy_loss) #G
optimizer.zero_grad()
total_loss.backward() #H
torch.nn.utils.clip_grad_norm_(
model.parameters(), config.max_grad_norm) #I
optimizer.step()
The policy update weights action log-probabilities by the computed advantages, while a separate loss trains the value prediction. section 9.7




