A cooling controller’s immediate reward captures the cost of one decision, but its return captures what happens over the rest of the trajectory. Return is the discounted sum of rewards from a particular time step onward; discounting weights later rewards by successive powers of γ. section 9.7 section 9.2
For a trajectory with rewards rₜ, the return can be computed backward using Rₜ = rₜ + γRₜ₊₁. For a complete trajectory, the return beyond its final reward is zero. A rollout buffer can instead initialize from a final value estimate when the rollout has not ended. section 9.7
Cooling a data center
At each step, the cooling reward is the negative of energy cost plus temperature and pressure violation penalties. The return combines that reward with discounted later rewards, so it reflects the consequences of a sequence of cooling decisions rather than just one step. section 9.7
Listing 9.5 Reward computation with energy and violation penalties
energy_cost = (self.cfg.w_energy_fan *
(fan ** 2).mean(dim=1) +
self.cfg.w_energy_valve *
(valve ** 2).mean(dim=1)) #A
temp_penalty = soft_hinge_penalty(
self.temp, self.cfg.temp_min, self.cfg.temp_max,
self.cfg.softplus_beta).mean(dim=1) *
self.cfg.w_temp_violation #B
pressure_penalty = soft_hinge_penalty(
self.dp, self.cfg.dp_min, self.cfg.dp_max,
self.cfg.softplus_beta).mean(dim=1) *
self.cfg.w_pressure_violation #C
rewards = -(energy_cost + temp_penalty +
pressure_penalty) #D
Inspect the energy and violation penalties that make up each cooling reward before the rewards are combined into returns. section 9.7
A return describes an observed trajectory segment; expected return averages performance across possible trajectories. Policy gradients aim to increase that expected performance. In the cooling example, subtracting the critic’s state-value prediction from the observed return gives an advantage: how much better or worse the segment was than expected. Advantages are normalized before the policy update for stability. section 9.7 section 9.2
