liveBook · Manning

REINFORCE

As explained in Applied Reinforcement Learning

A Monte Carlo policy-gradient algorithm that updates policy parameters using sampled returns and the gradient of the log probability of selected actions.

A data center must balance cooling energy against the risk of overheating. Fan speeds and valve openings can vary continuously, so choosing from a small set of fixed actions may sacrifice precision. REINFORCE addresses this kind of decision by learning a policy—a probability distribution over actions—directly from sampled experience. section 9.0 section 9.3

REINFORCE is a Monte Carlo policy-gradient algorithm. It samples trajectories under the current policy, measures the rewards that follow each action, and adjusts policy parameters by gradient ascent. The gradient of the log probability of an action points toward making that action more likely; the sampled return determines the direction and strength of the update. section 9.3 section 9.8

A vanilla REINFORCE update follows three steps:

  • Collect complete trajectories by sampling actions from the current policy and observing rewards. section 9.3
  • For each action, compute the return-to-go: the sum of rewards from that time onward, optionally discounted by γ. section 9.3
  • Multiply each action’s log-probability gradient by its return-to-go, average the resulting gradients across sampled trajectories, and update the policy by gradient ascent. section 9.3

The policy gradient theorem makes this possible without knowing the environment’s transition dynamics. Vanilla REINFORCE uses complete sampled returns rather than bootstrapping, and needs neither a replay buffer nor a target network. section 9.2 section 9.3

A single cooling fan

Suppose a policy models fan speed with a Gaussian distribution. At a temperature of 25°C and rising, it outputs a mean of 0.7 and a standard deviation of 0.2, then samples a speed of 0.82. If the return-to-go from that action is −50, the update weights its log-probability gradient negatively, pushing against making that sampled action more likely in that state. section 9.3

A practical weakness is high variance: a small batch of trajectories can produce a misleading update. A baseline reduces this noise by replacing the return-to-go Gₜ with Gₜ − b(sₜ). A state-value estimate V(sₜ) supplies a baseline for comparing the observed outcome with what was expected from that state. This difference is an advantage: positive values reinforce actions relative to the baseline, while negative values push against them. A separate value network, or critic, can learn the baseline while the actor updates the policy. section 9.3 section 9.4

Coordinating eight cooling zones

In a simulated data center, each of eight interconnected zones contributes eight observed features, including temperature, workload, and neighboring temperature. The policy selects fan speed and valve opening for each zone: 16 continuous controls in total. Rewards penalize energy use and temperature or pressure violations. A Transformer-based actor-critic uses attention to coordinate the zones and a value baseline to support policy updates. section 9.7 section 9.8

Listing 9.10 Policy gradient update with value baseline and entropy regularization
data = buffer.get_tensors()
T, B = returns.shape
obs_flat = data['observations'].view(
    T * B, config.n_zones, -1)                        #A
actions_flat = data['actions'].view(
    T * B, config.n_zones, 2)  #A
returns_flat = returns.view(T * B)
advantages_flat = advantages.view(T * B)
 
returns_mean = returns_flat.mean()
returns_std = returns_flat.std() + 1e-8
returns_normalized = (
    returns_flat - returns_mean) / returns_std  #B
 
log_probs, entropy, values = \
    model.evaluate_actions(obs_flat, actions_flat)  #C
 
policy_loss = -(
    log_probs * advantages_flat.detach()).mean()  #D
 
value_loss = F.mse_loss(
    values, returns_normalized.detach())  #E
 
entropy_loss = -entropy.mean()  #F
 
total_loss = (policy_loss +
    config.value_coef * value_loss +
    config.entropy_coef * entropy_loss)  #G
 
optimizer.zero_grad()
total_loss.backward()  #H
torch.nn.utils.clip_grad_norm_(
    model.parameters(), config.max_grad_norm) #I
optimizer.step()

Inspect a policy-gradient update using advantages, a value loss, and entropy regularization. section 9.7

How Applied Reinforcement Learning covers it

How deep it goes
Works through sampled gradients and return-to-go, then develops value baselines and an implemented Transformer actor-critic.
Examples
Illustrates updates with a Gaussian fan policy and implements an eight-zone data center cooling simulation.
What you can do after
Builds a REINFORCE-based cooling controller with a Transformer policy, value baseline, and entropy regularization.
How it connects
Connects policy gradients with Monte Carlo returns, state-value baselines, actor-critic methods, attention, and entropy regularization.

Also explained in

Compare how 2 books explain REINFORCE →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.