liveBook · Manning

Q(lambda)

As explained in Applied Reinforcement Learning

Q(lambda) is an off-policy action-value learning algorithm that combines Q-learning targets with eligibility traces and resets traces after non-greedy actions.

Q(λ) combines Q-learning’s off-policy target with eligibility traces, spreading learning from a reward across recently visited state-action pairs. Watkins’ cut resets those traces when the agent takes a non-greedy action, preventing credit from propagating through exploratory choices. section 6.6

Its update logic:

  • Use the maximum next-state Q-value to form the off-policy target. section 6.6
  • Propagate the temporal-difference error through eligible state-action pairs; clear traces after a non-greedy action. section 6.6

Fuel ordering

In gas-station inventory scheduling, orders can take days to affect outcomes. Eligibility traces let the agent assign delayed reward to earlier ordering decisions. section 6.6

Listing 6.15 Q(λ) agent implementation (Watkins' Q(λ))
class QLearningEligibilityAgent(EligibilityTraceAgent):
    def __init__(self, action_space_size: int, learning_rate: float, 
             discount_factor: float, lambda_param: float, 
             epsilon_start: float, epsilon_decay: float):
    super().__init__(action_space_size, "Q(λ)", learning_rate, 
                    discount_factor, lambda_param, epsilon_start, epsilon_decay)
 
    def update(self, state: EnvironmentState, action: int, reward: float,
               next_state: EnvironmentState, next_action: int, done: bool):
        state_key = state.to_tuple()
        next_state_key = next_state.to_tuple()
        
        current_q = self.q_table[state_key][action]
        if done:
            target = reward
        else:
            next_max_q = np.max(
            [CA]self.q_table[next_state_key]) #A
            target = reward + self.discount_factor * next_max_q
        td_error = target - current_q
        
        self.eligibility_traces[state_key][action] = 1.0
        
        self.update_traces_and_q_values(td_error)
        
        greedy_next_action = np.argmax(
                    [CA]self.q_table[next_state_key]) #B
        if next_action != greedy_next_action:
            self.eligibility_traces.clear() #C

Inspect the tabular agent implementation of Watkins’ Q(λ). section 6.6

How Applied Reinforcement Learning covers it

How deep it goes
Explains the off-policy target, trace updates, and Watkins’ cut, then shows an agent implementation.
Examples
Uses gas-station fuel ordering, where decisions take days to produce rewards.
What you can do after
Build a tabular Q(λ) agent that updates eligible pairs and resets traces after non-greedy actions.
How it connects
Connects Q-learning, eligibility traces, Watkins’ cut, and SARSA(λ).

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.