liveBook · Manning

Off-policy method

As explained in Applied Reinforcement Learning

Off-policy learning is reinforcement learning in which a target policy is evaluated or improved using experience generated by a potentially different behavior policy.

An off-policy method learns about a target policy using experience generated by a potentially different behavior policy. This lets an agent explore actions while learning a decision rule that favors the best actions currently estimated. section 6.3

The contrast is on-policy learning: there, the agent learns about the same policy it uses to act. Both approaches can update estimates incrementally from experience, but they differ in which policy the update reflects. section 6.3

Exploring while learning with Q-learning

Suppose an agent selects actions with an ε-greedy behavior policy, sometimes exploring instead of choosing its current best action. After observing a reward and next state, Q-learning updates the current action value using the highest estimated action value in that next state. The update therefore targets greedy future choices, even if the agent’s next actual action is exploratory. SARSA instead uses the value of the action actually selected next. section 6.4.2

Figure 6.6 Q-learning Algorithm Flowchart.

Follow the Q-learning flow from action selection and observation to the greedy-value update. section 6.4.2

Off-policy learning can also reuse experience from older policies. For example, DQN stores transitions in a replay buffer and uses them repeatedly for learning. Off-policy is not the same distinction as online versus offline: the former concerns which policy generated the experience; the latter concerns when learning occurs. section 12.1 section 6.7

The separation is useful for exploration and data reuse, but the learning target matters. Q-learning’s maximum-value update assumes the best estimated action will be taken next; that optimism can be misleading when estimates are inaccurate or future execution differs. section 6.3 section 6.4.2

How Applied Reinforcement Learning covers it

How deep it goes
Explains policy separation and walks through Q-learning’s update, then identifies replay in DQN without detailing its implementation.
Examples
Uses ε-greedy Q-learning and DQN’s replay buffer to illustrate exploratory behavior and reuse of older transitions.
What it assumes
Assumes prior familiarity with generalized policy iteration while introducing temporal-difference learning and the on-policy distinction.
What you can do after
Evaluate and understand when exploratory experience or older transitions can support learning about a different policy.
How it connects
Connects temporal-difference bootstrapping, exploration and exploitation, and generalized policy iteration.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.