liveBook · Manning

Replacing trace

As explained in Applied Reinforcement Learning

A replacing trace sets the trace of a visited state or state-action pair to a fixed maximum value instead of accumulating repeated increments.

When a state is revisited, a replacing trace sets its eligibility to 1 instead of adding another increment to its decayed value. This prevents repeated visits from pushing the trace above 1 and limits overcounting; it is described as more stable in off-policy learning. section 6.5

Reset after decay

If a state's trace decays to 0.36 before the state is visited again, a replacing trace sets it to 1. An accumulating trace would instead raise it to 1.36. section 6.5

How Applied Reinforcement Learning covers it

How deep it goes
Defines the reset rule and contrasts it with accumulating traces using a numerical example.
Examples
Uses a trace decaying to 0.36 before a revisit, then compares reset to 1 with accumulation to 1.36.
What you can do after
Understand how a revisited state's trace resets after decay rather than accumulating another increment.
How it connects
Contrasts with accumulating traces and relates eligibility traces to TD(λ) and off-policy learning.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.