An off-policy method learns about a target policy using experience generated by a potentially different behavior policy. This lets an agent explore actions while learning a decision rule that favors the best actions currently estimated. section 6.3
The contrast is on-policy learning: there, the agent learns about the same policy it uses to act. Both approaches can update estimates incrementally from experience, but they differ in which policy the update reflects. section 6.3
Exploring while learning with Q-learning
Suppose an agent selects actions with an ε-greedy behavior policy, sometimes exploring instead of choosing its current best action. After observing a reward and next state, Q-learning updates the current action value using the highest estimated action value in that next state. The update therefore targets greedy future choices, even if the agent’s next actual action is exploratory. SARSA instead uses the value of the action actually selected next. section 6.4.2
Figure 6.6 Q-learning Algorithm Flowchart.
Follow the Q-learning flow from action selection and observation to the greedy-value update. section 6.4.2
Off-policy learning can also reuse experience from older policies. For example, DQN stores transitions in a replay buffer and uses them repeatedly for learning. Off-policy is not the same distinction as online versus offline: the former concerns which policy generated the experience; the latter concerns when learning occurs. section 12.1 section 6.7
The separation is useful for exploration and data reuse, but the learning target matters. Q-learning’s maximum-value update assumes the best estimated action will be taken next; that optimism can be misleading when estimates are inaccurate or future execution differs. section 6.3 section 6.4.2
