Convergence is the settling of successive value estimates or policies toward a stable result. Depending on the method, that result is checked against an optimal solution or signaled by training metrics that stop changing meaningfully. section 6.2 section 11.6.5
Look for convergence in different ways:
- In temporal-difference learning, compare learned state values with analytically optimal values; experience can approach the target without known transition probabilities. section 6.2
- In policy evaluation, repeat Bellman updates until value changes become negligible. section 4.4
- In GRPO training, a stable KL alongside accuracy that stops improving over 20–30 iterations signals an equilibrium for that configuration. section 11.6.5
Restaurant scheduling
For a restaurant deciding when to seat a party, temporal-difference estimates for four hours remaining reach $85.17 against an optimal $85.47; the one-hour estimate matches the $50 optimum. section 6.2
Figure 6.3 Convergence behavior of Temporal difference learning algorithm.
The convergence plot makes the temporal-difference learning trend visible. section 6.2





