liveBook · Manning

On-policy method

An on-policy method learns or evaluates a policy using experience generated by that policy or its current version.

How 2 Manning books explain it

Applied Reinforcement Learning

On-policy learning is reinforcement learning in which the policy generating experience is the same policy being evaluated or improved.

How it is used
Distinguishes learning from the current decision policy from learning about a separate target policy.
How deep it goes
Defines the distinction, walks through SARSA's decision process, and surveys experience sources in deep reinforcement learning.
Examples
Uses SARSA's state-action-reward-state-action tuple and a portfolio policy selecting among eight allocations.
What you can do after
Evaluate and understand whether an update learns from the policy generating experience or from a different target policy.

Read how Applied Reinforcement Learning explains on-policy method →

Reinforcement Learning from Human Feedback

On-policy optimization, or on-policy learning, updates a policy using data generated by that same policy or its current update distribution.

How it is used
Uses current-policy generations to supply feedback for language-model reinforcement learning and closely related checkpoints to collect preference data.
How deep it goes
Explains sampling and synchronization trade-offs, preference-data usage, and an empirical forgetting mechanism without deriving a policy-gradient objective.
Examples
Illustrates the method with actor-learner GPU systems and Tülu 3’s multistage post-training recipe.
What you can do after
Evaluate the freshness and relevance of training data while weighing synchronization cost, stability, and potential forgetting.

Read how Reinforcement Learning from Human Feedback explains on-policy method →

Also covered in

What to know first

Related concepts

Questions about on-policy method

What is on-policy method?

An on-policy method learns or evaluates a policy using experience generated by that policy or its current version.

Which Manning books explain on-policy method?

Applied Reinforcement Learning and Reinforcement Learning from Human Feedback.

What should I know before learning on-policy method?

Manning books that teach on-policy method usually introduce policy, reward signal, state transition and action first.

What is the difference between on-policy method and off-policy method?

An on-policy method learns or evaluates a policy using experience generated by that policy or its current version. An off-policy method learns or evaluates a target policy using experience generated by a different behavior policy.

What kinds of on-policy method are there?

Kinds of on-policy method include SARSA(λ) and Proximal Policy Optimization.

What is on-policy method part of?

On-policy method is part of Tülu 3 training recipe. The Tülu 3 training recipe is a three-stage post-training configuration combining instruction tuning, on-policy preference optimization, and reinforcement learning with verifiable rewards.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.