liveBook · Manning

Bias-variance tradeoff

As explained in Applied Reinforcement Learning

The bias-variance tradeoff is the balance between systematic estimation error and variability across samples or outcomes.

In sequential learning, TD(0) offers fast updates, while Monte Carlo accounts for longer-term outcomes. The bias–variance tradeoff describes the balance between these approaches; TD(λ) makes it adjustable by averaging n-step returns. section 6.5

TD(λ) shifts the balance through λ:

  • At λ=1, it is close to Monte Carlo; intermediate values, such as 0.7, blend the approaches. section 6.5
  • Eligibility traces provide an online backward-view implementation, assigning more credit to recently visited states and letting older traces decay. section 6.5
Figure 6.7 Comparison of TD(0), Monte Carlo and TD(λ) methods.

Compare TD(0), Monte Carlo, and TD(λ) to see the methods positioned along this tradeoff. section 6.5

How Applied Reinforcement Learning covers it

How deep it goes
Explains the TD(λ) return blend, λ settings, and online eligibility traces without deriving the algorithms.
Examples
Uses λ=0.7 as an intermediate blend and repeated visits followed by a large reward to illustrate credit assignment.
What you can do after
Apply TD(λ) by tuning λ to blend n-step returns and balance bias against variance.
How it connects
Connects reinforcement learning with TD(0), Monte Carlo, n-step TD, TD(λ), and eligibility traces.

Also explained in

Compare how 5 books explain bias-variance tradeoff →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.