liveBook · Manning

Policy gradient method

A policy gradient method is a reinforcement-learning method that optimizes a parameterized policy by estimating the gradient of expected return with respect to its parameters.

How 6 Manning books explain it

Applied Reinforcement Learning

A reinforcement-learning method that directly optimizes the parameters of a policy using the gradient of expected return.

How it is used
Motivates direct policy optimization for continuous, high-dimensional control and learned randomized decisions.
How deep it goes
Develops the policy gradient objective and REINFORCE update, then implements an actor-critic cooling simulation.
Examples
Uses fan-speed trajectories, randomized patrol routes, and an eight-zone data center cooling simulation.
What you can do after
Build a simulated cooling controller using a Gaussian policy, value baseline, and Transformer actor-critic.

Read how Applied Reinforcement Learning explains policy gradient method →

Reinforcement Learning from Human Feedback

Policy gradient is a reinforcement-learning technique that optimizes policy parameters by estimating gradients of expected reward from sampled trajectories.

How it is used
Uses policy gradients to translate reward-model feedback into language-model behavior during RLHF.
How deep it goes
Derives the trajectory gradient with the log-derivative trick, then develops baselines, algorithm variants, and practical loss implementations.
Examples
Uses masked completion log probabilities and an advantage-weighted loss, alongside a clipped PPO loss example.
What you can do after
Evaluate/understand how reward signals, baselines, and clipping influence policy updates and their stability.

Read how Reinforcement Learning from Human Feedback explains policy gradient method →

Optimization Algorithms: AI techniques for design, planning, and control problems

A policy gradient is a reinforcement-learning method that optimizes a policy by estimating gradients of expected return with respect to policy parameters.

How it is used
Uses policy probabilities and reward feedback to learn reinforcement-learning policies and optimize predicted TSP tours.
How deep it goes
Explains policy probabilities, expected return, discounting, and applications, but stops short of deriving gradient estimates or training code.
Examples
Uses the Reach the Treasure trajectory and a traveling salesperson tour-length objective as concrete examples.
What you can do after
Apply policy gradients to frame reward-based decision learning and minimize predicted tour length in TSP models.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains policy gradient method →

Build a DeepSeek Model (From Scratch)

A policy-gradient loss is an optimization objective that weights policy log probabilities by feedback-derived advantages to update a policy.

How it is used
Updates sampled LLM token probabilities from response-level rewards during reinforcement learning.
How deep it goes
Derives a reward-weighted log-probability update and flags masks, old-policy probabilities, and KL penalties as implementation concerns.
Examples
Uses stochastic next-token choices and a verifier-scored answer of 11 to illustrate trajectory-level feedback.
What you can do after
Understand how sampled-token log probabilities and rewards form a policy update.

Read how Build a DeepSeek Model (From Scratch) explains policy gradient method →

LLM Evaluation and Alignment, The Foundational Ideas

A policy-gradient method optimizes a parameterized action-selection policy by estimating how changes to its parameters affect expected reward.

How it is used
Uses policy-gradient updates to optimize generation systems against automated reward metrics.
How deep it goes
Contrasts metric-optimized outputs with human judgments but does not detail the policy-gradient algorithm.
Examples
Compares translation with BLEU, summarization with ROUGE, and image captioning with CIDEr.
What you can do after
Understand why a rising reward score may not indicate better human-rated outputs.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains policy gradient method →

Build a Reasoning Model (From Scratch)

The reinforcement-learning objective used to adjust a policy so actions with better outcomes become more likely and worse ones less likely.

How it is used
Drives the GRPO policy update by turning relative rollout feedback into a training objective.
How deep it goes
Defines the loss, shows the sign flip, and explains detach; it stops short of the later weight update implementation.
Examples
Uses a chef analogy, a worked sign-flip example, and a PyTorch listing with stacked logps.
What you can do after
Apply it via PyTorch to compute a minimizable loss that increases high-advantage rollout likelihoods.

Read how Build a Reasoning Model (From Scratch) explains policy gradient method →

What to know first

Related concepts

Questions about policy gradient method

What is policy gradient method?

A policy gradient method is a reinforcement-learning method that optimizes a parameterized policy by estimating the gradient of expected return with respect to its parameters.

Which Manning books explain policy gradient method?

Applied Reinforcement Learning, Reinforcement Learning from Human Feedback, Optimization Algorithms: AI techniques for design, planning, and control problems, Build a DeepSeek Model (From Scratch), LLM Evaluation and Alignment, The Foundational Ideas and Build a Reasoning Model (From Scratch).

What should I know before learning policy gradient method?

Manning books that teach policy gradient method usually introduce gradient, reinforcement learning, reward signal and gradient descent first.

What is the difference between policy gradient method and Group Relative Policy Optimization?

A policy gradient method is a reinforcement-learning method that optimizes a parameterized policy by estimating the gradient of expected return with respect to its parameters. Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.

What kinds of policy gradient method are there?

Kinds of policy gradient method include Proximal Policy Optimization and REINFORCE.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.