A gas station must decide whether to order fuel now or wait, balancing uncertain demand against holding costs and stockouts. SARSA(λ) learns such sequential decisions from experience: it combines on-policy SARSA updates with eligibility traces, which help assign credit to earlier decisions as new outcomes arrive. section 6.5 section 6.6
SARSA stands for State-Action-Reward-State-Action. It learns the value Q(s,a) of an action in a state using the reward and the next action actually selected by its current policy. An ε-greedy policy usually chooses a high-value action but sometimes explores another. This makes SARSA on-policy: it learns about the same behavior it uses to act. section 6.4.1 section 6.4
At each step, SARSA(λ) extends that decision loop:
- Take an action, observe the reward and next state, and select the next action using the current policy. section 6.4.1
- Calculate the temporal-difference error: δ = r + γQ(s′,a′) − Q(s,a). At a terminal state, the target is the reward alone. section 6.4 section 6.6
- Use eligibility traces to distribute the learning signal to recently visited state-action pairs, then let their influence decay as the agent continues. section 6.5
The parameter λ controls how far credit extends beyond the latest step. This offers a middle ground between one-step temporal-difference learning, which updates immediately, and Monte Carlo learning, which waits for a complete episode. Higher λ moves the treatment toward longer returns; it does not remove the need to manage the trade-off between bootstrapping and variability. section 6.5
Fuel ordering
Represent a state by current inventory, the total quantity on order, and the time until the next delivery. An action chooses an order quantity; the daily reward accounts for sales, stockouts, holding costs, and ordering costs. As demand and deliveries unfold, SARSA(λ) can update the values of recent ordering decisions rather than crediting only the latest one. section 6.6 section 6.5
Listing 6.14 SARSA(λ) agent implementation
class SARSAEligibilityAgent(EligibilityTraceAgent):
def __init__(self, action_space_size: int, learning_rate: float,
discount_factor: float, lambda_param: float,
epsilon_start: float, epsilon_decay: float):
super().__init__(action_space_size, "SARSA(λ)", learning_rate,
discount_factor,
[CA]lambda_param, epsilon_start, epsilon_decay)
def update(self, state: EnvironmentState, action: int, reward: float,
next_state: EnvironmentState, next_action: int, done: bool):
state_key = state.to_tuple()
next_state_key = next_state.to_tuple()
current_q = self.q_table[state_key][action]
if done:
target = reward
else:
next_q = self.q_table[next_state_key][next_action] #A
target = reward + self.discount_factor * next_q
td_error = target - current_q
self.eligibility_traces[state_key][action] = 1.0 #B
self.update_traces_and_q_values(td_error) #C
Inspect the agent’s on-policy target and its handling of terminal states. section 6.6
Unlike off-policy Q-learning, SARSA uses the next action selected by its behavior policy rather than the best hypothetical next action. That distinction matters when exploration itself carries risk: the learned values reflect the actions the agent actually takes. section 6.4 section 6.4.1
