The score function, ∇θ log πθ(a|s), is the gradient of an action’s log-probability with respect to policy parameters. It points in a direction that makes that action more likely in that state. Log-probabilities also turn products of probabilities into sums and keep very small probabilities numerically manageable. section 9.2
A fan-speed choice
If a policy assigns a 30% probability to fan speed 0.82, its log-probability is about −1.20. The score function indicates how to adjust the policy parameters to make that choice more likely. In a policy-gradient update, a positive return reinforces the actions in a trajectory; a negative return suppresses them. section 9.2 section 9.8
In the policy-gradient theorem, score functions are weighted by trajectory returns and summed over time. Taking the log of trajectory probability makes the expression a sum; environment-dynamics terms vanish when differentiated with respect to policy parameters. This lets the gradient be computed without knowing those dynamics. section 9.2 section 9.8

