liveBook · Manning

Accumulating trace

As explained in Applied Reinforcement Learning

An accumulating trace increases by a fixed increment when its associated state or state-action pair is visited, in addition to retaining its decayed value.

An accumulating trace tracks eligibility for credit in TD(λ). Each step, traces decay by γλ; when a state is visited, add 1 to its decayed trace. Repeated visits can raise a trace above 1, giving frequently visited states stronger credit. section 6.5

A repeated visit

With γ = 0.9 and λ = 0.8, a trace of 0.5 decays to 0.36. Visiting that state adds 1, making its trace 1.36. section 6.5

Figure 6.8 Update mechanism of temporal difference learning with eligibility traces.

See the eligibility-trace update mechanism in temporal-difference learning. section 6.5

How Applied Reinforcement Learning covers it

How deep it goes
Works through the decay-and-increment rule numerically and contrasts accumulating traces with replacing traces.
Examples
Uses a γ = 0.9, λ = 0.8 calculation to show a trace rising from 0.5 to 1.36.
What you can do after
Understand how decay and repeated visits determine an accumulating trace's value.
How it connects
Connects eligibility traces with TD(λ), replacing traces, and state-action values.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.