liveBook · Manning

Score function

As explained in Applied Reinforcement Learning

The gradient of the log probability of an action, indicating how parameter changes affect that action's likelihood.

The score function, ∇θ log πθ(a|s), is the gradient of an action’s log-probability with respect to policy parameters. It points in a direction that makes that action more likely in that state. Log-probabilities also turn products of probabilities into sums and keep very small probabilities numerically manageable. section 9.2

A fan-speed choice

If a policy assigns a 30% probability to fan speed 0.82, its log-probability is about −1.20. The score function indicates how to adjust the policy parameters to make that choice more likely. In a policy-gradient update, a positive return reinforces the actions in a trajectory; a negative return suppresses them. section 9.2 section 9.8

In the policy-gradient theorem, score functions are weighted by trajectory returns and summed over time. Taking the log of trajectory probability makes the expression a sum; environment-dynamics terms vanish when differentiated with respect to policy parameters. This lets the gradient be computed without knowing those dynamics. section 9.2 section 9.8

How Applied Reinforcement Learning covers it

How deep it goes
Defines the gradient and explains its direction, log-probability motivation, and cancellation of environment-dynamics terms.
Examples
Illustrates the calculation with a 30% probability of choosing fan speed 0.82.
What you can do after
Apply score-function directions to reinforce actions from positive-return trajectories and suppress actions from negative-return trajectories.
How it connects
Connects score functions to policy gradients, REINFORCE, and trajectory returns.

Also explained in

Compare how 2 books explain score function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.