Entropy regularization adds a bonus for keeping a policy’s action distribution spread out. This encourages exploration and helps prevent the policy from committing to one action before it has explored alternatives. The objective can include the term βH(πθ), where β is a small coefficient, typically 0.01. section 9.4 section 9.8
A Gaussian policy for fan speed
With a mean of 0.5, a Gaussian policy with σ = 0.3 samples a broad range of fan speeds; with σ = 0.01, it almost always chooses 0.5. The broader distribution has higher entropy and explores more widely. section 9.4
In PPO, the entropy bonus is subtracted from the loss so the policy does not settle on one response too quickly. section 10.7
