Q(λ) combines Q-learning’s off-policy target with eligibility traces, spreading learning from a reward across recently visited state-action pairs. Watkins’ cut resets those traces when the agent takes a non-greedy action, preventing credit from propagating through exploratory choices. section 6.6
Its update logic:
- Use the maximum next-state Q-value to form the off-policy target. section 6.6
- Propagate the temporal-difference error through eligible state-action pairs; clear traces after a non-greedy action. section 6.6
Fuel ordering
In gas-station inventory scheduling, orders can take days to affect outcomes. Eligibility traces let the agent assign delayed reward to earlier ordering decisions. section 6.6
Listing 6.15 Q(λ) agent implementation (Watkins' Q(λ))
class QLearningEligibilityAgent(EligibilityTraceAgent):
def __init__(self, action_space_size: int, learning_rate: float,
discount_factor: float, lambda_param: float,
epsilon_start: float, epsilon_decay: float):
super().__init__(action_space_size, "Q(λ)", learning_rate,
discount_factor, lambda_param, epsilon_start, epsilon_decay)
def update(self, state: EnvironmentState, action: int, reward: float,
next_state: EnvironmentState, next_action: int, done: bool):
state_key = state.to_tuple()
next_state_key = next_state.to_tuple()
current_q = self.q_table[state_key][action]
if done:
target = reward
else:
next_max_q = np.max(
[CA]self.q_table[next_state_key]) #A
target = reward + self.discount_factor * next_max_q
td_error = target - current_q
self.eligibility_traces[state_key][action] = 1.0
self.update_traces_and_q_values(td_error)
greedy_next_action = np.argmax(
[CA]self.q_table[next_state_key]) #B
if next_action != greedy_next_action:
self.eligibility_traces.clear() #C
Inspect the tabular agent implementation of Watkins’ Q(λ). section 6.6
