liveBook · Manning

Policy representation advantage

As explained in Applied Reinforcement Learning

Policy representation advantage is the benefit of directly learning a compact decision rule when the corresponding value function is more complex.

Sometimes the best decision rule is simpler than the value function used to derive it. A value function must assign a value to every state; the policy can instead express a compact rule for choosing actions. When that rule is simpler to learn directly, optimizing the policy can be more efficient. section 9.1

Navigating a maze

A maze’s value function must capture how far each cell is from the exit along the best path, and walls can make nearby cells have very different values. Yet a useful policy might be as simple as “if there’s an opening to the right, go right; otherwise go down.” section 9.1

Direct policy learning is not always preferable: policy-gradient methods can have higher-variance estimates, converge to local optima, and use samples less efficiently than DQN. They can fit continuous control, high-dimensional action spaces, and settings where stochastic policies are desirable. section 9.1

Figure 9.1 Value-Based vs. Policy-Based Reinforcement Learning.

Compare the value-based and policy-based approaches. section 9.1

How Applied Reinforcement Learning covers it

How deep it goes
Explains the idea with a maze example without deriving policy-gradient methods.
Examples
Contrasts a maze’s complex value landscape with a compact action rule and mentions data-center cooling.
What you can do after
Evaluate when direct policy learning may fit, considering action spaces, stochasticity, and method limitations.
How it connects
Links policy-gradient methods and stochastic policies to value functions and value-based methods.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.