Policy gradient method
A policy gradient method is a reinforcement-learning method that optimizes a parameterized policy by estimating the gradient of expected return with respect to its parameters.
Applied Reinforcement Learning
A reinforcement-learning method that directly optimizes the parameters of a policy using the gradient of expected return.
- How it is used
- Motivates direct policy optimization for continuous, high-dimensional control and learned randomized decisions.
- How deep it goes
- Develops the policy gradient objective and REINFORCE update, then implements an actor-critic cooling simulation.
- Examples
- Uses fan-speed trajectories, randomized patrol routes, and an eight-zone data center cooling simulation.
- What you can do after
- Build a simulated cooling controller using a Gaussian policy, value baseline, and Transformer actor-critic.
Read how Applied Reinforcement Learning explains policy gradient method →
Reinforcement Learning from Human Feedback
Policy gradient is a reinforcement-learning technique that optimizes policy parameters by estimating gradients of expected reward from sampled trajectories.
- How it is used
- Uses policy gradients to translate reward-model feedback into language-model behavior during RLHF.
- How deep it goes
- Derives the trajectory gradient with the log-derivative trick, then develops baselines, algorithm variants, and practical loss implementations.
- Examples
- Uses masked completion log probabilities and an advantage-weighted loss, alongside a clipped PPO loss example.
- What you can do after
- Evaluate/understand how reward signals, baselines, and clipping influence policy updates and their stability.
Read how Reinforcement Learning from Human Feedback explains policy gradient method →
Optimization Algorithms: AI techniques for design, planning, and control problems
A policy gradient is a reinforcement-learning method that optimizes a policy by estimating gradients of expected return with respect to policy parameters.
- How it is used
- Uses policy probabilities and reward feedback to learn reinforcement-learning policies and optimize predicted TSP tours.
- How deep it goes
- Explains policy probabilities, expected return, discounting, and applications, but stops short of deriving gradient estimates or training code.
- Examples
- Uses the Reach the Treasure trajectory and a traveling salesperson tour-length objective as concrete examples.
- What you can do after
- Apply policy gradients to frame reward-based decision learning and minimize predicted tour length in TSP models.
Build a DeepSeek Model (From Scratch)
A policy-gradient loss is an optimization objective that weights policy log probabilities by feedback-derived advantages to update a policy.
- How it is used
- Updates sampled LLM token probabilities from response-level rewards during reinforcement learning.
- How deep it goes
- Derives a reward-weighted log-probability update and flags masks, old-policy probabilities, and KL penalties as implementation concerns.
- Examples
- Uses stochastic next-token choices and a verifier-scored answer of 11 to illustrate trajectory-level feedback.
- What you can do after
- Understand how sampled-token log probabilities and rewards form a policy update.
Read how Build a DeepSeek Model (From Scratch) explains policy gradient method →
LLM Evaluation and Alignment, The Foundational Ideas
A policy-gradient method optimizes a parameterized action-selection policy by estimating how changes to its parameters affect expected reward.
- How it is used
- Uses policy-gradient updates to optimize generation systems against automated reward metrics.
- How deep it goes
- Contrasts metric-optimized outputs with human judgments but does not detail the policy-gradient algorithm.
- Examples
- Compares translation with BLEU, summarization with ROUGE, and image captioning with CIDEr.
- What you can do after
- Understand why a rising reward score may not indicate better human-rated outputs.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains policy gradient method →
Build a Reasoning Model (From Scratch)
The reinforcement-learning objective used to adjust a policy so actions with better outcomes become more likely and worse ones less likely.
- How it is used
- Drives the GRPO policy update by turning relative rollout feedback into a training objective.
- How deep it goes
- Defines the loss, shows the sign flip, and explains detach; it stops short of the later weight update implementation.
- Examples
- Uses a chef analogy, a worked sign-flip example, and a PyTorch listing with stacked logps.
- What you can do after
- Apply it via PyTorch to compute a minimizable loss that increases high-advantage rollout likelihoods.
Read how Build a Reasoning Model (From Scratch) explains policy gradient method →
What to know first
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- reward signalA reward signal is a value or feedback indicating the desirability, benefit, or cost associated with an action, outcome, or behavior.
- gradient descentGradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
Related concepts
- Proximal Policy Optimizationincludes · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- log probabilityuses · A log probability is the logarithm, usually the natural logarithm, of a probability assigned to an event or outcome.
- advantage functionuses · An advantage function measures the expected return of an action relative to a baseline, typically the expected return under a policy at the same state.
- REINFORCEincludes · REINFORCE is a Monte Carlo policy-gradient algorithm that updates policy parameters using sampled returns and the gradient of selected actions’ log probabilities.
- reward signaluses · A reward signal is a value or feedback indicating the desirability, benefit, or cost associated with an action, outcome, or behavior.
- Group Relative Policy Optimizationcontrasts with · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.
Questions about policy gradient method
What is policy gradient method?
A policy gradient method is a reinforcement-learning method that optimizes a parameterized policy by estimating the gradient of expected return with respect to its parameters.
Which Manning books explain policy gradient method?
Applied Reinforcement Learning, Reinforcement Learning from Human Feedback, Optimization Algorithms: AI techniques for design, planning, and control problems, Build a DeepSeek Model (From Scratch), LLM Evaluation and Alignment, The Foundational Ideas and Build a Reasoning Model (From Scratch).
What should I know before learning policy gradient method?
Manning books that teach policy gradient method usually introduce gradient, reinforcement learning, reward signal and gradient descent first.
What is the difference between policy gradient method and Group Relative Policy Optimization?
A policy gradient method is a reinforcement-learning method that optimizes a parameterized policy by estimating the gradient of expected return with respect to its parameters. Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.
What kinds of policy gradient method are there?
Kinds of policy gradient method include Proximal Policy Optimization and REINFORCE.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





