The log-derivative trick rewrites a probability gradient as ∇θP(x) = P(x)∇θlog P(x). For a trajectory, taking the log turns its product of probabilities into a sum. Terms for the initial state and environment transitions do not depend on θ, so only policy terms remain; the gradient no longer requires known environment dynamics. section 9.2
Figure 9.3 Mathematical proof of Policy Gradient Theorem.
Follow the policy-gradient derivation that applies the identity to trajectory probabilities. section 9.2
