liveBook · Manning

Entropy

As explained in Applied Reinforcement Learning

Entropy quantifies the uncertainty or randomness of a probability distribution.

Entropy measures how spread out a probability distribution is: high entropy means greater uncertainty, while low entropy means probability is concentrated on one action. In policy-gradient training, entropy regularization rewards policies for staying uncertain enough to explore, reducing the risk of committing to a strategy too early. The objective adds a bonus, βH(πθ), to expected return; β is typically 0.01. section 9.4

A Gaussian action policy

For a policy with mean 0.5, σ = 0.3 gives a broad distribution over fan speeds and high entropy; σ = 0.01 concentrates actions near 0.5 and has low entropy. Entropy often falls as a policy gains confidence, but falling too quickly can prevent it from exploring better alternatives. section 9.4

How Applied Reinforcement Learning covers it

How deep it goes
Defines entropy intuitively, compares Gaussian policies, and gives a regularized objective with a typical coefficient.
Examples
Compares Gaussian policies with mean 0.5 and standard deviations 0.3 and 0.01.
What you can do after
Apply an entropy bonus to encourage exploration while training a policy.
How it connects
Connects entropy to probability distributions, policy gradients, exploration, and premature convergence.

Also explained in

Compare how 19 books explain entropy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.