liveBook · Manning

N-step TD

As explained in Applied Reinforcement Learning

n-step TD is a temporal-difference method whose target uses rewards and state estimates across n future steps.

n-step TD bridges one-step TD and Monte Carlo learning by using outcomes across multiple future steps. TD(λ) combines n-step returns: λ tunes the balance between bias and variance, with λ=1 approaching Monte Carlo. Computing these returns requires future rewards and states, while eligibility traces provide an online backward-view implementation of TD(λ). section 6.5

Figure 6.7 Comparison of TD(0), Monte Carlo and TD(λ) methods.

Compare the roles of TD(0), Monte Carlo, and TD(λ). section 6.5

How Applied Reinforcement Learning covers it

How deep it goes
Explains the n-step-return blend and its future-data requirement, then points to eligibility traces for online updates.
What you can do after
Understand how TD(λ) trades bias against variance and relates to online eligibility-trace updates.
How it connects
Connects temporal-difference learning with Monte Carlo, TD(0), TD(λ), and eligibility traces.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.