A portfolio manager must choose an allocation while still exploring alternatives. An on-policy method learns to improve the same decision rule, or policy, that generates its experience. If that policy sometimes explores, learning reflects those exploratory choices too. section 6.3 section 12.1
This differs from off-policy learning, where a behavior policy generates experience while the agent learns about a different target policy. For example, the behavior policy might explore while the target policy favors the best-valued action. On-policy learning instead evaluates and improves the policy currently being used. section 6.3
SARSA: learn from the action actually chosen
SARSA updates an action value using a five-part experience: current state, action, reward, next state, and next action. The next action is chosen under the current policy, often ε-greedy: usually select the highest-valued action, but sometimes choose randomly. The update uses the value of that chosen action, rather than assuming the agent will take the best-valued one. section 6.4.1
Figure 6.5 SARSA Algorithm Flowchart.
Follow the SARSA flow from action selection through observation and updating. section 6.4.1
The source of experience also shapes deep reinforcement learning. REINFORCE, PPO, and GRPO use data generated by the current policy; the described implementations collect fresh trajectories, or fresh groups of answers for GRPO, and discard them after an update. By contrast, off-policy DQN can reuse older transitions from a replay buffer. section 12.1

