liveBook · Manning

On-policy method

As explained in Applied Reinforcement Learning

On-policy learning is reinforcement learning in which the policy generating experience is the same policy being evaluated or improved.

A portfolio manager must choose an allocation while still exploring alternatives. An on-policy method learns to improve the same decision rule, or policy, that generates its experience. If that policy sometimes explores, learning reflects those exploratory choices too. section 6.3 section 12.1

This differs from off-policy learning, where a behavior policy generates experience while the agent learns about a different target policy. For example, the behavior policy might explore while the target policy favors the best-valued action. On-policy learning instead evaluates and improves the policy currently being used. section 6.3

SARSA: learn from the action actually chosen

SARSA updates an action value using a five-part experience: current state, action, reward, next state, and next action. The next action is chosen under the current policy, often ε-greedy: usually select the highest-valued action, but sometimes choose randomly. The update uses the value of that chosen action, rather than assuming the agent will take the best-valued one. section 6.4.1

Figure 6.5 SARSA Algorithm Flowchart.

Follow the SARSA flow from action selection through observation and updating. section 6.4.1

The source of experience also shapes deep reinforcement learning. REINFORCE, PPO, and GRPO use data generated by the current policy; the described implementations collect fresh trajectories, or fresh groups of answers for GRPO, and discard them after an update. By contrast, off-policy DQN can reuse older transitions from a replay buffer. section 12.1

How Applied Reinforcement Learning covers it

How deep it goes
Defines the distinction, walks through SARSA's decision process, and surveys experience sources in deep reinforcement learning.
Examples
Uses SARSA's state-action-reward-state-action tuple and a portfolio policy selecting among eight allocations.
What you can do after
Evaluate and understand whether an update learns from the policy generating experience or from a different target policy.
How it connects
Connects policy evaluation and improvement with temporal-difference learning, exploration, and off-policy learning.

Also explained in

Compare how 2 books explain on-policy method →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.