liveBook · Manning

Convergence

As explained in Applied Reinforcement Learning

Convergence is the approach of successive numerical estimates or policies toward a stable limiting result.

Convergence is the settling of successive value estimates or policies toward a stable result. Depending on the method, that result is checked against an optimal solution or signaled by training metrics that stop changing meaningfully. section 6.2 section 11.6.5

Look for convergence in different ways:

  • In temporal-difference learning, compare learned state values with analytically optimal values; experience can approach the target without known transition probabilities. section 6.2
  • In policy evaluation, repeat Bellman updates until value changes become negligible. section 4.4
  • In GRPO training, a stable KL alongside accuracy that stops improving over 20–30 iterations signals an equilibrium for that configuration. section 11.6.5

Restaurant scheduling

For a restaurant deciding when to seat a party, temporal-difference estimates for four hours remaining reach $85.17 against an optimal $85.47; the one-hour estimate matches the $50 optimum. section 6.2

Figure 6.3 Convergence behavior of Temporal difference learning algorithm.

The convergence plot makes the temporal-difference learning trend visible. section 6.2

How Applied Reinforcement Learning covers it

How deep it goes
Defines convergence and gives numerical and metric-based checks, but does not provide a formal proof.
Examples
Uses restaurant table scheduling, resource-allocation value updates, and KL and accuracy trends during training.
What you can do after
Evaluate convergence by comparing learned values with optimal values or checking whether KL and accuracy have stabilized.
How it connects
Connects convergence to temporal-difference learning, dynamic programming, policy evaluation, and GRPO.

Also explained in

Compare how 6 books explain convergence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.