On-policy method
An on-policy method learns or evaluates a policy using experience generated by that policy or its current version.
Applied Reinforcement Learning
On-policy learning is reinforcement learning in which the policy generating experience is the same policy being evaluated or improved.
- How it is used
- Distinguishes learning from the current decision policy from learning about a separate target policy.
- How deep it goes
- Defines the distinction, walks through SARSA's decision process, and surveys experience sources in deep reinforcement learning.
- Examples
- Uses SARSA's state-action-reward-state-action tuple and a portfolio policy selecting among eight allocations.
- What you can do after
- Evaluate and understand whether an update learns from the policy generating experience or from a different target policy.
Read how Applied Reinforcement Learning explains on-policy method →
Reinforcement Learning from Human Feedback
On-policy optimization, or on-policy learning, updates a policy using data generated by that same policy or its current update distribution.
- How it is used
- Uses current-policy generations to supply feedback for language-model reinforcement learning and closely related checkpoints to collect preference data.
- How deep it goes
- Explains sampling and synchronization trade-offs, preference-data usage, and an empirical forgetting mechanism without deriving a policy-gradient objective.
- Examples
- Illustrates the method with actor-learner GPU systems and Tülu 3’s multistage post-training recipe.
- What you can do after
- Evaluate the freshness and relevance of training data while weighing synchronization cost, stability, and potential forgetting.
Read how Reinforcement Learning from Human Feedback explains on-policy method →
Also covered in
What to know first
- policyA policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.
- reward signalA reward signal is a value or feedback indicating the desirability, benefit, or cost associated with an action, outcome, or behavior.
- state transitionA state transition is a change from one defined state of a system, object, or process to another according to an action, event, condition, or rule.
- actionAn action is a choice or operation selected or performed by an actor that can affect a system, environment, or process.
Related concepts
- SARSA(λ)includes · SARSA(λ) is an on-policy SARSA variant that uses eligibility traces to propagate temporal-difference credit to previously visited state-action pairs.
- Proximal Policy Optimizationincludes · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- off-policy methodcontrasts with · An off-policy method learns or evaluates a target policy using experience generated by a different behavior policy.
- policy gradient methodused by · A policy gradient method is a reinforcement-learning method that optimizes a parameterized policy by estimating the gradient of expected return with respect to its parameters.
Questions about on-policy method
What is on-policy method?
An on-policy method learns or evaluates a policy using experience generated by that policy or its current version.
Which Manning books explain on-policy method?
Applied Reinforcement Learning and Reinforcement Learning from Human Feedback.
What should I know before learning on-policy method?
Manning books that teach on-policy method usually introduce policy, reward signal, state transition and action first.
What is the difference between on-policy method and off-policy method?
An on-policy method learns or evaluates a policy using experience generated by that policy or its current version. An off-policy method learns or evaluates a target policy using experience generated by a different behavior policy.
What kinds of on-policy method are there?
Kinds of on-policy method include SARSA(λ) and Proximal Policy Optimization.
What is on-policy method part of?
On-policy method is part of Tülu 3 training recipe. The Tülu 3 training recipe is a three-stage post-training configuration combining instruction tuning, on-policy preference optimization, and reinforcement learning with verifiable rewards.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

