liveBook · Manning

Chapter 9: The Calculus of Decisions: Policy Gradient Methods

What you will learn in chapter 9 of Applied Reinforcement Learning

This introduction motivates direct policy optimization using the challenges of continuous, coupled data center cooling control.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction motivates direct policy optimization using the challenges of continuous, coupled data center cooling control.

Key terms: cooling zones, continuous, state space, action space, coupled, Deep Q-Networks, Q-values, discretizing, combinatorial explosion, policy gradient methods

9.1 Why optimizing the policy directly?

This section explains why directly optimizing a policy can be advantageous over value-based methods, while acknowledging its tradeoffs.

Key terms: discrete action bottleneck, Q-value, discretize, continuous controls, stochastic policy, probability distribution, policy gradient methods, principled exploration, ε-greedy exploration, policy representation advantage

9.2 The mathematics of policy gradients

This section develops the policy gradient objective and theorem, explaining how trajectory returns guide policy updates without a model of environment dynamics.

Key terms: objective function, trajectory, discounted return, gradient ascent, learning rate, policy gradient theorem, trajectory probability, log-derivative trick, environment dynamics, score function

9.3 The REINFORCE algorithm

This section explains how REINFORCE estimates policy gradients from sampled trajectories and updates continuous or discrete action policies.

Key terms: Monte Carlo estimate, trajectories, unbiased, variance, return-to-go, discounted sum, credit assignment, REINFORCE, gradient ascent, log-policy gradient

9.4 Variance reduction through baselines

This section explains how baselines, a learned critic, advantage normalization, and entropy regularization make REINFORCE training more stable and exploratory.

Key terms: REINFORCE, return-to-go, baseline, variance, unbiased gradient, state-value function, advantage, actor, policy network, critic

9.5 Attention mechanism: learning what to look at

This section explains how attention relates interacting cooling zones through query-key-value computations and multi-head attention.

Key terms: attention mechanism, representations, attention weights, adaptive feature extraction, scalability, Query-Key-Value, query, key, value, dot product

9.6 The Transformer architecture

This section explains Transformer encoder components, positional encoding, and their use in coordinating data center cooling decisions.

Key terms: multi-head self-attention, position-wise feedforward network, Transformer encoder layer, residual connection, skip connection, layer normalization, positional encoding, permutation-invariant, spatial proximity, sinusoidal functions

9.7 Case study: data center cooling optimization

This section models data center cooling as a coordinated control problem and applies a Transformer-based REINFORCE agent, then compares it with a conventional controller.

Key terms: cooling zones, ring topology, state space, action space, thermal coupling, thermal mass, reward function, energy cost, temperature violations, pressure violations

9.8 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 8: Deep Q-Networks for High-dimensional Data Chapter 10: Fine-tuning large language models with PPO →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.