Dynamic programming plans in a Markov decision process (MDP) when the environment’s transition probabilities and rewards are known. Rather than learning from sampled experience, it uses this complete model to compute behavior through Bellman equations. section 4.0 section 4.1
Two iterative algorithms refine values and policies:
- Value iteration recursively updates value functions toward convergence. section 4.1
- Policy iteration recursively improves policies and value functions until convergence. section 4.1
Resource allocation and scale
A resource-allocation problem with a budget of 12 and four projects has 1,053 states. Raising the budget to 100 and the number of projects to 10 produces nearly six million states, illustrating how computation can grow. section 4.5
Figure 4.1 Five different paradigms in solving Markov decision process.
Compare dynamic programming with other paradigms for solving MDPs. section 4.1
The method’s main constraints are large state spaces and the need for a complete, accurate model. That makes it difficult to scale to complex real-world problems, while its Bellman-based ideas provide a foundation for reinforcement learning methods that work without a full model. section 4.5 section 6.7







