Chapter 9: The Calculus of Decisions: Policy Gradient Methods
This introduction motivates direct policy optimization using the challenges of continuous, coupled data center cooling control.
After this chapter you will be able to
- DQN discretization limitations. Explain why a discrete-action DQN becomes impractical for the cooling problem despite handling high-dimensional states.
- Direct policy optimization. Explain how policy gradient methods approach decisions differently from choosing actions using Q-values.
- Discrete Action Bottleneck. Explain why discretizing continuous, high-dimensional controls makes a DQN-style action representation impractical.
- Stochastic Policy Exploration. Distinguish an action selected from learned values from an action sampled from a stochastic policy, and explain why learned randomness can help.
- Policy Representation Advantage. Explain how an optimal policy can be simpler to represent than the corresponding value function, using the maze example.
- Policy objective and ascent. Explain what the policy gradient objective measures and how gradient ascent adjusts policy parameters to improve it.
- Policy gradient theorem derivation. Explain why the policy gradient can be expressed without knowing environment transition dynamics, including the role of the log-derivative trick.
- Score function and return. Explain how the score function and trajectory return combine to guide policy improvement, using the fan-speed example if helpful.
- Monte Carlo gradient estimation. Explain how sampled trajectories estimate the policy gradient and distinguish an unbiased estimate from a low-variance one.
- Return to go credit. Explain what return-to-go measures and why REINFORCE uses it instead of the full trajectory return for an action.
- REINFORCE update cycle. Describe the REINFORCE cycle and explain how the return changes the direction and strength of an action's policy update.
- Gaussian parameter adaptation. Trace how a Gaussian policy adjusts its mean and standard deviation in response to a sampled action's outcome, using the fan example.
- Discrete softmax policy. Explain how REINFORCE represents and updates a policy over discrete actions, contrasting its distribution with the continuous Gaussian policy.
- Baseline variance reduction. Explain why vanilla REINFORCE can have a noisy, scale-dependent gradient and how subtracting a state-dependent baseline changes the signal.
- State value and advantage. Apply the state-value baseline to calculate an advantage and explain how its sign affects an action's policy update.
- Actor critic training roles. Describe how the actor and critic are trained in REINFORCE with a learned baseline.
- Entropy regularization and loss. Explain how entropy regularization supports exploration and how it appears alongside policy and critic losses.
- Attention for Zone Coordination. Explain why the cooling policy needs attention across zones and how attention produces context-sensitive zone representations.
- Query Key Value Computation. Trace how query-key scores become weights and how those weights combine values to form a zone's output.
- Multi Head Attention Architecture. Describe how multi-head attention captures different zone relationships and combines the results.
- Transformer Encoder Data Flow. Trace how multi-head self-attention, position-wise feedforward networks, and stacked encoder layers build representations of interacting zones.
- Residual Connections and Normalization. Explain how residual connections and layer normalization wrap each encoder sub-layer and support training.
- Position and Input Order. Explain why zone representations need positional encoding before attention and how it helps the model use spatial arrangement.
- Cooling Model Output Heads. Describe how the data center Transformer uses zone observations and produces policy and value outputs for coordinated cooling.
- Cooling Control MDP. Describe the cooling environment’s observations, continuous controls, and coupled dynamics. Explain why decisions must coordinate zones over time.
- Safety and Energy Reward. Explain how the reward balances energy use against temperature and pressure violations, and why the implementation uses a smooth temperature penalty.
- Transformer Actor Critic Architecture. Describe how zone observations become Transformer inputs and how the actor and critic produce different outputs.
- On Policy Training Cycle. Trace the REINFORCE update from fresh parallel trajectories through returns, advantages, and the policy loss.
- Evaluation and Safety Tradeoff. Describe the reported training and comparison results, including the trade-off between energy use and temperature violations.
What it builds on
- DQN theory: Q-learning meets neural networks (chapter 8) · DQN uses a neural network to represent Q-values.
- Markov decision process (chapter 2) · Markov decision process states, actions and transition probabilities
- Tradeoff between exploitation and exploration (chapter 5) · The tradeoff between exploration and exploitation.
- Handle soft constrains with stepwise penalties (chapter 2) · Using stepwise penalties for constraints
Section by section
Introduction
This introduction motivates direct policy optimization using the challenges of continuous, coupled data center cooling control.
- Coupled cooling control problem: Describe the data center cooling problem, including its tradeoff and the interaction between zones.
Why it matters: This example shows why coordinating continuous decisions across zones is challenging. - DQN discretization limitations: Explain why a discrete-action DQN becomes impractical for the cooling problem despite handling high-dimensional states.
Why it matters: The limitations of discrete action choices motivate looking beyond DQN for continuous control. - Direct policy optimization: Explain how policy gradient methods approach decisions differently from choosing actions using Q-values.
Why it matters: This is the central shift from value-based action selection to directly learning a decision strategy.
Key terms: cooling zones, continuous, state space, action space, coupled, Deep Q-Networks, Q-values, discretizing, combinatorial explosion, policy gradient methods
9.1 Why optimizing the policy directly?
This section explains why directly optimizing a policy can be advantageous over value-based methods, while acknowledging its tradeoffs.
- Discrete Action Bottleneck: Explain why discretizing continuous, high-dimensional controls makes a DQN-style action representation impractical.
Why it matters: This explains why the cooling problem is a poor fit for a separate DQN output for every discretized action. - Stochastic Policy Exploration: Distinguish an action selected from learned values from an action sampled from a stochastic policy, and explain why learned randomness can help.
Why it matters: A policy that represents randomness directly can explore and avoid predictable behavior without relying on an external exploration heuristic. - Policy Representation Advantage: Explain how an optimal policy can be simpler to represent than the corresponding value function, using the maze example.
Why it matters: Direct policy learning can avoid representing a complicated value landscape when the decision rule itself is simple. - Policy Gradient Tradeoffs: Describe the limitations of policy gradient methods relative to value-based methods and identify when their advantages can justify using them.
Why it matters: Choosing policy gradients involves weighing their fit for the action space against their learning costs.
Key terms: discrete action bottleneck, Q-value, discretize, continuous controls, stochastic policy, probability distribution, policy gradient methods, principled exploration, ε-greedy exploration, policy representation advantage
9.2 The mathematics of policy gradients
This section develops the policy gradient objective and theorem, explaining how trajectory returns guide policy updates without a model of environment dynamics.
- Policy objective and ascent: Explain what the policy gradient objective measures and how gradient ascent adjusts policy parameters to improve it.
Why it matters: This objective tells us what policy gradient methods are trying to improve and how parameter updates are directed. - Policy gradient theorem derivation: Explain why the policy gradient can be expressed without knowing environment transition dynamics, including the role of the log-derivative trick.
Why it matters: The theorem lets an agent improve its policy without having to know how the environment transitions between states. - Score function and return: Explain how the score function and trajectory return combine to guide policy improvement, using the fan-speed example if helpful.
Why it matters: This connects the theorem's mathematical terms to learning from actual actions and outcomes.
Key terms: objective function, trajectory, discounted return, gradient ascent, learning rate, policy gradient theorem, trajectory probability, log-derivative trick, environment dynamics, score function
9.3 The REINFORCE algorithm
This section explains how REINFORCE estimates policy gradients from sampled trajectories and updates continuous or discrete action policies.
- Monte Carlo gradient estimation: Explain how sampled trajectories estimate the policy gradient and distinguish an unbiased estimate from a low-variance one.
Why it matters: Sampling makes the gradient practical to estimate, but its variability affects learning. - Return to go credit: Explain what return-to-go measures and why REINFORCE uses it instead of the full trajectory return for an action.
Why it matters: Credit is more useful when it excludes rewards that occurred before the action. - REINFORCE update cycle: Describe the REINFORCE cycle and explain how the return changes the direction and strength of an action's policy update.
Why it matters: This cycle connects observed outcomes to changes in the policy. - Gaussian policy representation: Describe how a Gaussian policy samples continuous actions and how its mean and standard deviation affect a sampled action's log-probability.
Why it matters: The distribution determines which continuous controls the policy tends to try. - Gaussian parameter adaptation: Trace how a Gaussian policy adjusts its mean and standard deviation in response to a sampled action's outcome, using the fan example.
Why it matters: The policy can adjust both its preferred control and how widely it explores. - Discrete softmax policy: Explain how REINFORCE represents and updates a policy over discrete actions, contrasting its distribution with the continuous Gaussian policy.
Why it matters: The same learning principle works for choices such as production levels as well as continuous controls.
Key terms: Monte Carlo estimate, trajectories, unbiased, variance, return-to-go, discounted sum, credit assignment, REINFORCE, gradient ascent, log-policy gradient
9.4 Variance reduction through baselines
This section explains how baselines, a learned critic, advantage normalization, and entropy regularization make REINFORCE training more stable and exploratory.
- Baseline variance reduction: Explain why vanilla REINFORCE can have a noisy, scale-dependent gradient and how subtracting a state-dependent baseline changes the signal.
Why it matters: A baseline makes the learning signal easier to distinguish without changing the expected policy gradient. - State value and advantage: Apply the state-value baseline to calculate an advantage and explain how its sign affects an action's policy update.
Why it matters: Advantage shows whether an action's outcome deserves reinforcement relative to what was expected from its state. - Actor critic training roles: Describe how the actor and critic are trained in REINFORCE with a learned baseline.
Why it matters: The two networks divide action selection and learning the expected return used as a baseline. - Batch advantage normalization: Describe how advantages are normalized across a batch and why this helps training.
Why it matters: Normalization provides another way to stabilize updates when the scale of advantages changes. - Entropy regularization and loss: Explain how entropy regularization supports exploration and how it appears alongside policy and critic losses.
Why it matters: Maintaining exploration helps the policy avoid committing to a poor strategy too early.
Key terms: REINFORCE, return-to-go, baseline, variance, unbiased gradient, state-value function, advantage, actor, policy network, critic
9.5 Attention mechanism: learning what to look at
This section explains how attention relates interacting cooling zones through query-key-value computations and multi-head attention.
- Attention for Zone Coordination: Explain why the cooling policy needs attention across zones and how attention produces context-sensitive zone representations.
Why it matters: Coordinated cooling decisions depend on what is happening across related zones. - Learned Attention Benefits: Describe how attention learns useful zone relationships and what its weights and shared mechanism provide.
Why it matters: These properties make attention adaptable to different zone relationships and provide insight into its coordination. - Query Key Value Computation: Trace how query-key scores become weights and how those weights combine values to form a zone's output.
Why it matters: This computation is how attention turns relevance between zones into information for the policy. - Dot Product Scaling: Explain why attention scales query-key dot products before softmax.
Why it matters: Scaling helps keep the attention computation trainable when key vectors have many dimensions. - Multi Head Attention Architecture: Describe how multi-head attention captures different zone relationships and combines the results.
Why it matters: Different heads can capture several kinds of zone interaction in one representation.
Key terms: attention mechanism, representations, attention weights, adaptive feature extraction, scalability, Query-Key-Value, query, key, value, dot product
9.6 The Transformer architecture
This section explains Transformer encoder components, positional encoding, and their use in coordinating data center cooling decisions.
- Transformer Encoder Data Flow: Trace how multi-head self-attention, position-wise feedforward networks, and stacked encoder layers build representations of interacting zones.
Why it matters: This flow shows how a Transformer builds context-aware representations for decisions involving interacting entities. - Residual Connections and Normalization: Explain how residual connections and layer normalization wrap each encoder sub-layer and support training.
Why it matters: These two features help preserve information and keep deep networks trainable. - Position and Input Order: Explain why zone representations need positional encoding before attention and how it helps the model use spatial arrangement.
Why it matters: Cooling interactions depend on where zones are, not just on their sensor readings. - Sinusoidal Position Representation: Describe how sinusoidal positional encodings at different frequencies help represent zone positions and relative offsets.
Why it matters: Multi-scale position signals help the model learn spatial patterns without memorizing every pair of zones. - Cooling Model Output Heads: Describe how the data center Transformer uses zone observations and produces policy and value outputs for coordinated cooling.
Why it matters: The two heads turn shared cross-zone representations into cooling choices and a baseline value estimate.
Key terms: multi-head self-attention, position-wise feedforward network, Transformer encoder layer, residual connection, skip connection, layer normalization, positional encoding, permutation-invariant, spatial proximity, sinusoidal functions
9.7 Case study: data center cooling optimization
This section models data center cooling as a coordinated control problem and applies a Transformer-based REINFORCE agent, then compares it with a conventional controller.
- Cooling Control MDP: Describe the cooling environment’s observations, continuous controls, and coupled dynamics. Explain why decisions must coordinate zones over time.
Why it matters: The environment’s interconnected zones and delayed responses explain why cooling is a sequential coordination problem. - Safety and Energy Reward: Explain how the reward balances energy use against temperature and pressure violations, and why the implementation uses a smooth temperature penalty.
Why it matters: The reward tells the agent when extra cooling energy is worthwhile for safety. - Transformer Actor Critic Architecture: Describe how zone observations become Transformer inputs and how the actor and critic produce different outputs.
Why it matters: The network can coordinate local controls while evaluating the state of the entire facility. - Bounded Stochastic Action Sampling: Explain how the policy samples continuous actions and transforms them into valid control settings.
Why it matters: The policy needs both valid physical settings and correctly computed probabilities for learning. - On Policy Training Cycle: Trace the REINFORCE update from fresh parallel trajectories through returns, advantages, and the policy loss.
Why it matters: This cycle shows how the agent turns cooling experience into a better policy. - Evaluation and Safety Tradeoff: Describe the reported training and comparison results, including the trade-off between energy use and temperature violations.
Why it matters: The results illustrate that a better reward can mean spending more energy to avoid costly safety violations.
Key terms: cooling zones, ring topology, state space, action space, thermal coupling, thermal mass, reward function, energy cost, temperature violations, pressure violations
9.8 Summary
Where this is used later
- Overview of Large Language Models Architecture (chapter 10) · Direct policy optimization provides context for treating a model as a policy.
- Mapping Language Model to a Markov Decision Process (chapter 10) · A policy maps states to actions or action probabilities.
- The common structure of deep reinforcement learning algorithms (chapter 12) · Direct policy optimization
- Proximal Policy Optimization: From REINFORCE to PPO (chapter 10) · PPO builds on policy optimization.
- Stage 3: PPO fine-tuning (chapter 10) · The policy gradient theorem uses policy probabilities.
- Drug discovery and molecular design (chapter 11) · REINFORCE as a policy-based reinforcement learning algorithm
- Operations research and combinatorial optimization (chapter 11) · attention mechanisms
- Stage 2: reward model training (chapter 10) · The transformer architecture and its attention-based representations.
Concepts explained in this chapter
- REINFORCEA Monte Carlo policy-gradient algorithm that updates policy parameters using sampled returns and the gradient of the log probability of selected actions.
- policy gradient methodA reinforcement-learning method that directly optimizes the parameters of a policy using the gradient of expected return.
- advantage functionAdvantage measures how much an action's expected return exceeds or falls below the value of its state under a policy.
- transformer architectureA neural-network architecture built around self-attention, position-wise feedforward transformations, residual connections, and normalization.
- returnThe discounted sum of future rewards from a particular time step in a trajectory.
- cooling zoneA monitored region of a data center with local thermal and workload conditions that influence cooling controls.
- log probabilityThe logarithm of the probability or probability density assigned to an outcome by a probability model.
- transformer encoderA neural-network encoder formed from stacked self-attention and feedforward layers with residual connections and normalization.
- multi-head attentionMulti-head attention performs several independent attention computations in parallel and combines their outputs to capture different relationship patterns.
- softmaxSoftmax converts a vector of real-valued scores into nonnegative probabilities that sum to one.
- policy gradient theoremA theorem expressing the gradient of expected policy return using policy score functions and action values without requiring differentiation of the environment dynamics.
- reparameterization trickA stochastic-gradient technique that expresses a random sample as a deterministic transformation of parameter-independent noise.
- sigmoidA differentiable function that maps real numbers to values between zero and one.
- softplus functionA smooth function, log(1 + exp(x)), commonly used to provide differentiable positive-valued outputs or penalties.
- log-derivative trickA calculus identity that rewrites derivatives of probability distributions as expectations involving derivatives of their logarithms.
- score functionThe gradient of the log probability of an action, indicating how parameter changes affect that action's likelihood.
- trajectoryA trajectory is a sampled sequence of states, actions, and outcomes generated by an agent interacting with an environment.
- criticThe value-estimating component of an actor-critic system that predicts expected returns and supplies a learning baseline.
- entropy regularizationEntropy regularization adds an entropy-based term to the objective to encourage diversity in a policy's action distribution.
- gradient ascentAn iterative optimization method that updates parameters in the direction of a function's gradient to increase its value.
- objective functionA mathematical quantity that a learning or optimization procedure seeks to minimize or maximize.
- positional embeddingA representation added to inputs to encode their positions or identities so attention models can use ordering or spatial relationships.
- residual connectionA neural-network connection that adds a layer's input directly to its transformed output.
- attentionAn attention mechanism computes input-dependent weights to combine representations according to their relevance to one another.
- Monte Carlo simulationMonte Carlo estimation approximates a quantity by averaging values obtained from random samples.
- categorical distributionA categorical distribution assigns probabilities to a finite set of mutually exclusive outcomes.
- discrete action spaceA discrete action space is a finite set of distinct actions from which an agent selects.
- logitA logit is an unnormalized real-valued score that softmax transforms into an action probability.
- normal distributionA Gaussian distribution is a continuous probability distribution characterized by a mean and standard deviation.
- reward-to-goReturn-to-go is the discounted sum of rewards from a specified decision time onward in a trajectory.
- standard deviationStandard deviation is the square root of variance and measures the typical spread of a probability distribution.
- advantage normalizationAdvantage normalization rescales a batch of advantage estimates to have zero mean and unit variance.
- entropyEntropy quantifies the uncertainty or randomness of a probability distribution.
- queryA query is an attention vector representing the information an input element seeks from other elements.
- query-key-value frameworkThe query-key-value framework represents each input with query, key, and value vectors to compute attention-weighted outputs.
- valueA value is an attention vector containing the information contributed by an input element after attention weights are computed.
- Transformer encoder layerA Transformer encoder block containing self-attention and position-wise feedforward sub-layers with residual connections and normalization.
- layer normalizationA normalization technique that rescales and recenters activations within each layer to stabilize neural-network computation.
- permutation invarianceThe property that reordering inputs changes the output only by the corresponding reordering, without changing relational results.
- policy headA policy head is the output component of a policy model that converts an internal representation into action decisions or action parameters.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
