A policy head converts a model’s internal representation into action distributions. In a Transformer-based policy, it produces a distribution for each zone; a separate value head estimates the state value used as a baseline. section 9.6
Figure 9.8 Transformer architecture.
The architecture diagram shows where the policy head and value head sit after the Transformer layers. section 9.6
