liveBook · Manning

Log-derivative trick

As explained in Applied Reinforcement Learning

A calculus identity that rewrites derivatives of probability distributions as expectations involving derivatives of their logarithms.

The log-derivative trick rewrites a probability gradient as ∇θP(x) = P(x)∇θlog P(x). For a trajectory, taking the log turns its product of probabilities into a sum. Terms for the initial state and environment transitions do not depend on θ, so only policy terms remain; the gradient no longer requires known environment dynamics. section 9.2

Figure 9.3 Mathematical proof of Policy Gradient Theorem.

Follow the policy-gradient derivation that applies the identity to trajectory probabilities. section 9.2

How Applied Reinforcement Learning covers it

How deep it goes
Derives the identity and follows its application from trajectory probabilities to policy-only gradient terms.
Examples
Uses a trajectory probability built from initial-state, policy-action, and state-transition probabilities as its algebraic example.
What you can do after
Understand how to rewrite probability gradients and eliminate unknown transition dynamics from a policy-gradient calculation.
How it connects
Connects policy gradients to the score function and REINFORCE by removing environment-transition terms from the gradient.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.