liveBook · Manning

Chapter 4: Perfect Knowledge, Optimal Policy: Dynamic Programming

What you will learn in chapter 4 of Applied Reinforcement Learning

This introduction contrasts model-based and model-free methods and situates dynamic programming as a recursive approach to solving Markov decision processes with a known model.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction contrasts model-based and model-free methods and situates dynamic programming as a recursive approach to solving Markov decision processes with a known model.

Key terms: model-based methods, model-free methods, transition function, reward function, sampled transitions, dynamic programming, overlapping subproblems, recursive components, stochastic environments, transition and reward information

4.1 Paradigms on solving Markov decision process

This section compares five decision-making paradigms by how they use rewards, experience, models, and simulated futures.

Key terms: Multi-Armed Bandits, exploration-exploitation trade-off, temporal difference learning, bootstrapping, value estimates, Monte Carlo learning, episode, full return, variance, dynamic programming

4.2 The domino decision rule: Bellman equations

This section explains Bellman equations as recursive ways to evaluate states and actions under a policy, and relates their values to the advantage function.

Key terms: Bellman equations, immediate reward, discounted expected value, domino rule, state-value function, Vπ(s), policy, discount factor, backup diagram, action-value function

4.3 Solving bellman equations: Generalized Policy Iteration

This section explains generalized policy iteration and demonstrates policy evaluation and improvement with a simplified inventory problem.

Key terms: Generalized Policy Iteration, Policy Evaluation, Policy Improvement, Bellman expectation equations, greedy, discounted value, state, action, transition dynamics, deterministic demand

4.4 Hands-on code: solving a resource allocation problem

This section models a startup’s resource allocation problem as an MDP, solves it with generalized policy iteration, and interprets the resulting policy.

Key terms: budget remaining, project status vector, action space, do nothing, action constraints, transition probability function, base success probability, enhancement factor, allocation, reward function

4.5 Limitations of dynamic programming

This section explains why dynamic programming is difficult to apply at scale despite its foundational role in reinforcement learning.

Key terms: curse of a massive state space, value function, state, curse of a perfect model, transition probabilities, reward function, planning algorithm, model-free methods, generalized policy iteration, Bellman equations

4.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 3: Design Custom Environments for Reinforcement Learning Algorithms Chapter 5: Contextual Bandit: Optimizing Stochastic One-step Decisions →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.