liveBook · Manning

Policy gradient theorem

As explained in Applied Reinforcement Learning

A theorem expressing the gradient of expected policy return using policy score functions and action values without requiring differentiation of the environment dynamics.

The policy gradient theorem shows how to improve a parameterized decision policy without differentiating through unknown environment dynamics. Instead of estimating action values and choosing an argmax, the agent adjusts the policy directly. section 9.0 section 9.2

Its gradient weights each policy score, ∇θ log πθ(aₜ|sₜ), by the trajectory return: ∇θJ(θ) = E[Σₜ ∇θ log πθ(aₜ|sₜ) R(τ)]. Positive returns reinforce actions; negative returns suppress them. section 9.8

Cooling control

For data-center cooling, fan speeds create continuous control choices. The theorem supports learning a policy for these choices without discretizing the action space or knowing the cooling system’s transition dynamics. section 9.0 section 9.8

Figure 9.3 Mathematical proof of Policy Gradient Theorem.

Trace how the trajectory-probability factorization and log-derivative steps isolate the policy-dependent terms. section 9.2

How Applied Reinforcement Learning covers it

How deep it goes
Derives the trajectory-likelihood and log-derivative steps that remove unknown transition probabilities from the gradient.
Examples
Uses continuous fan-speed control and coordinated data-center cooling as motivating examples.
What it assumes
Uses a few calculus and Markov decision process concepts to develop the theorem’s proof.
What you can do after
Understand how score functions and returns form a policy gradient without modeling environment dynamics.
How it connects
Connects policy optimization to REINFORCE and Markov decision processes.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.