liveBook · Manning

Entropy regularization

As explained in Applied Reinforcement Learning

Entropy regularization adds an entropy-based term to the objective to encourage diversity in a policy's action distribution.

Entropy regularization adds a bonus for keeping a policy’s action distribution spread out. This encourages exploration and helps prevent the policy from committing to one action before it has explored alternatives. The objective can include the term βH(πθ), where β is a small coefficient, typically 0.01. section 9.4 section 9.8

A Gaussian policy for fan speed

With a mean of 0.5, a Gaussian policy with σ = 0.3 samples a broad range of fan speeds; with σ = 0.01, it almost always chooses 0.5. The broader distribution has higher entropy and explores more widely. section 9.4

In PPO, the entropy bonus is subtracted from the loss so the policy does not settle on one response too quickly. section 10.7

How Applied Reinforcement Learning covers it

How deep it goes
Defines entropy through distribution spread, gives a Gaussian policy example, and specifies an entropy bonus in the objective.
Examples
Compares Gaussian fan-speed policies with σ = 0.3 and σ = 0.01 to illustrate differing exploration.
What you can do after
Understand how an entropy bonus can discourage premature commitment to a narrow action distribution.
How it connects
Connects entropy regularization to policy gradients, REINFORCE, PPO, and exploration.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.