Policy gradient method

As explained in Optimization Algorithms: AI techniques for design, planning, and control problems

A policy gradient is a reinforcement-learning method that optimizes a policy by estimating gradients of expected return with respect to policy parameters.

A policy gradient method optimizes a parameterized policy by estimating how changes to its parameters affect expected return. In reinforcement learning, the policy assigns action probabilities from observations, and training seeks a policy that maximizes cumulative discounted reward. section 12.1.2

Choosing a move in Reach the Treasure

Suppose an agent observes a situation and assigns 0.4 probability to moving left and 0.6 to moving right. Combining these action probabilities with possible rewards produces an expected return of 5.728 for one trajectory. Policy optimization uses this return-oriented objective to improve the action policy. section 12.1.2

The method connects policy choice with reward optimization:

• Represent decisions as probabilities over available actions given an observation. section 12.1.2
• Evaluate choices through expected cumulative discounted reward, with the discount factor controlling emphasis on immediate versus future rewards. section 12.1.2
• Use policy gradients alongside value iteration and policy iteration as iterative approaches to learning an effective policy. section 12.1.2
• Apply the approach to combinatorial optimization: a reinforcement-learning model can use policy gradients to minimize the predicted tour length for the traveling salesperson problem. section 11.7

Figure 12.4 Reach the Treasure game

Use the Reach the Treasure game to inspect how action probabilities, trajectories, and rewards combine into an expected return. section 12.1.2

How Optimization Algorithms: AI techniques for design, planning, and control problems covers it

How deep it goes

Explains policy probabilities, expected return, discounting, and applications, but stops short of deriving gradient estimates or training code.

Examples

Uses the Reach the Treasure trajectory and a traveling salesperson tour-length objective as concrete examples.

What it assumes

Assumes familiarity with policies, observations, actions, rewards, and trajectories while defining expected return and value functions.

What you can do after

Apply policy gradients to frame reward-based decision learning and minimize predicted tour length in TSP models.

How it connects

Connects reinforcement learning with policies, expected return, value functions, exploration, supervised learning, and traveling salesperson problem optimization.

liveBookmanning.com

Read Optimization Algorithms: AI techniques for design, planning, and control problems on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackPolicy gradient is a reinforcement-learning technique that optimizes policy parameters by estimating gradients of expected reward from sampled trajectories.
• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)The reinforcement-learning objective used to adjust a policy so actions with better outcomes become more likely and worse ones less likely.

Compare how 3 books explain policy gradient method →

Related concepts

• Proximal Policy Optimizationincludes · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
• log probabilityuses · A log probability is the logarithm, usually the natural logarithm, of a probability assigned to an event, outcome, token, or sequence.
• REINFORCEincludes · REINFORCE is a Monte Carlo policy-gradient algorithm that updates policy parameters using sampled returns and the gradient of selected actions’ log probabilities.
• reward signaluses · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.
• advantage functionuses · An advantage function in reinforcement learning estimates how much better or worse an action’s expected return is than a baseline value for the corresponding state or policy.
• reference modeluses · A reference model is a fixed policy distribution that serves as a baseline for comparing or constraining changes to an optimized policy.

This explainer was generated by Manning's knowledge graph from the text of Optimization Algorithms: AI techniques for design, planning, and control problems. The section links point to the parts of the book it draws on.