liveBook · Manning

Return

As explained in Applied Reinforcement Learning

The discounted sum of future rewards from a particular time step in a trajectory.

A cooling controller’s immediate reward captures the cost of one decision, but its return captures what happens over the rest of the trajectory. Return is the discounted sum of rewards from a particular time step onward; discounting weights later rewards by successive powers of γ. section 9.7 section 9.2

For a trajectory with rewards rₜ, the return can be computed backward using Rₜ = rₜ + γRₜ₊₁. For a complete trajectory, the return beyond its final reward is zero. A rollout buffer can instead initialize from a final value estimate when the rollout has not ended. section 9.7

Cooling a data center

At each step, the cooling reward is the negative of energy cost plus temperature and pressure violation penalties. The return combines that reward with discounted later rewards, so it reflects the consequences of a sequence of cooling decisions rather than just one step. section 9.7

Listing 9.5 Reward computation with energy and violation penalties
energy_cost = (self.cfg.w_energy_fan *
    (fan ** 2).mean(dim=1) +
    self.cfg.w_energy_valve *
    (valve ** 2).mean(dim=1))  #A
 
temp_penalty = soft_hinge_penalty(
    self.temp, self.cfg.temp_min, self.cfg.temp_max,
    self.cfg.softplus_beta).mean(dim=1) *
    self.cfg.w_temp_violation  #B
 
pressure_penalty = soft_hinge_penalty(
    self.dp, self.cfg.dp_min, self.cfg.dp_max,
    self.cfg.softplus_beta).mean(dim=1) *
    self.cfg.w_pressure_violation  #C
 
rewards = -(energy_cost + temp_penalty +
    pressure_penalty)  #D

Inspect the energy and violation penalties that make up each cooling reward before the rewards are combined into returns. section 9.7

A return describes an observed trajectory segment; expected return averages performance across possible trajectories. Policy gradients aim to increase that expected performance. In the cooling example, subtracting the critic’s state-value prediction from the observed return gives an advantage: how much better or worse the segment was than expected. Advantages are normalized before the policy update for stability. section 9.7 section 9.2

How Applied Reinforcement Learning covers it

How deep it goes
Defines discounted return and works through backward computation, critic-baseline subtraction, and advantage normalization.
Examples
Uses data center cooling rewards combining energy costs with temperature and pressure violation penalties.
What you can do after
Evaluate and understand how observed returns contribute to policy updates through an advantage baseline.
How it connects
Connects return to trajectories, policy gradients, value functions, and advantage estimation.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.