Sometimes the best decision rule is simpler than the value function used to derive it. A value function must assign a value to every state; the policy can instead express a compact rule for choosing actions. When that rule is simpler to learn directly, optimizing the policy can be more efficient. section 9.1
Navigating a maze
A maze’s value function must capture how far each cell is from the exit along the best path, and walls can make nearby cells have very different values. Yet a useful policy might be as simple as “if there’s an opening to the right, go right; otherwise go down.” section 9.1
Direct policy learning is not always preferable: policy-gradient methods can have higher-variance estimates, converge to local optima, and use samples less efficiently than DQN. They can fit continuous control, high-dimensional action spaces, and settings where stochastic policies are desirable. section 9.1
Figure 9.1 Value-Based vs. Policy-Based Reinforcement Learning.
Compare the value-based and policy-based approaches. section 9.1
