liveBook · Manning

SARSA(λ)

As explained in Applied Reinforcement Learning

SARSA(λ) is an on-policy temporal-difference algorithm that combines SARSA updates with eligibility traces.

A gas station must decide whether to order fuel now or wait, balancing uncertain demand against holding costs and stockouts. SARSA(λ) learns such sequential decisions from experience: it combines on-policy SARSA updates with eligibility traces, which help assign credit to earlier decisions as new outcomes arrive. section 6.5 section 6.6

SARSA stands for State-Action-Reward-State-Action. It learns the value Q(s,a) of an action in a state using the reward and the next action actually selected by its current policy. An ε-greedy policy usually chooses a high-value action but sometimes explores another. This makes SARSA on-policy: it learns about the same behavior it uses to act. section 6.4.1 section 6.4

At each step, SARSA(λ) extends that decision loop:

  • Take an action, observe the reward and next state, and select the next action using the current policy. section 6.4.1
  • Calculate the temporal-difference error: δ = r + γQ(s′,a′) − Q(s,a). At a terminal state, the target is the reward alone. section 6.4 section 6.6
  • Use eligibility traces to distribute the learning signal to recently visited state-action pairs, then let their influence decay as the agent continues. section 6.5

The parameter λ controls how far credit extends beyond the latest step. This offers a middle ground between one-step temporal-difference learning, which updates immediately, and Monte Carlo learning, which waits for a complete episode. Higher λ moves the treatment toward longer returns; it does not remove the need to manage the trade-off between bootstrapping and variability. section 6.5

Fuel ordering

Represent a state by current inventory, the total quantity on order, and the time until the next delivery. An action chooses an order quantity; the daily reward accounts for sales, stockouts, holding costs, and ordering costs. As demand and deliveries unfold, SARSA(λ) can update the values of recent ordering decisions rather than crediting only the latest one. section 6.6 section 6.5

Listing 6.14 SARSA(λ) agent implementation
class SARSAEligibilityAgent(EligibilityTraceAgent):
    def __init__(self, action_space_size: int, learning_rate: float, 
                 discount_factor: float, lambda_param: float, 
                 epsilon_start: float, epsilon_decay: float):
        super().__init__(action_space_size, "SARSA(λ)", learning_rate, 
                        discount_factor, 
                   [CA]lambda_param, epsilon_start, epsilon_decay)
    
    def update(self, state: EnvironmentState, action: int, reward: float,
               next_state: EnvironmentState, next_action: int, done: bool):
        state_key = state.to_tuple()
        next_state_key = next_state.to_tuple()
        
        current_q = self.q_table[state_key][action]
        if done:
            target = reward
        else:
            next_q = self.q_table[next_state_key][next_action] #A
            target = reward + self.discount_factor * next_q
        td_error = target - current_q
        
        self.eligibility_traces[state_key][action] = 1.0 #B
        
        self.update_traces_and_q_values(td_error) #C

Inspect the agent’s on-policy target and its handling of terminal states. section 6.6

Unlike off-policy Q-learning, SARSA uses the next action selected by its behavior policy rather than the best hypothetical next action. That distinction matters when exploration itself carries risk: the learned values reflect the actions the agent actually takes. section 6.4 section 6.4.1

How Applied Reinforcement Learning covers it

How deep it goes
Explains the TD(λ) trade-off and provides a tabular agent implementation with an on-policy target and terminal handling.
Examples
Uses a gas-station inventory simulation with fuel orders, uncertain demand, delayed deliveries, and stockout penalties.
What it assumes
Builds on temporal-difference learning, on-policy SARSA, action values, and eligibility traces.
What you can do after
Build it from scratch as a tabular agent for a simulated inventory problem.
How it connects
Connects SARSA, eligibility traces, temporal-difference learning, Monte Carlo learning, and Q-learning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.