liveBook · Manning

Chapter 2: Markov Decision Process: Turning Problems intro Solvable Models

What you will learn in chapter 2 of Applied Reinforcement Learning

This introduction frames Markov decision processes as a way to break down and reconnect complex business decisions over time.

After this chapter you will be able to

Section by section

Introduction

This introduction frames Markov decision processes as a way to break down and reconnect complex business decisions over time.

Key terms: systems thinking, analysis, synthesis, Markov decision process, uncertainty, dynamics

2.1 State: Anatomy of sequential decision making

This section introduces states as snapshots of a system and contrasts deterministic, known stochastic, and unknown stochastic transitions.

Key terms: state, sequential decision-making, deterministic transition system, stochastic systems with known transitions, transition probabilities, exploratory data analysis, stochastic systems with unknown transitions

2.2 Markov chain and Markov property

This section explains the Markov property, illustrates it with a simulated customer-state chain, and connects it to practical state design.

Key terms: Markov property, Markovian, curse of dimensionality, transitions, graph, transition matrix, simulation, empirical distribution, Buy Pro, Install Basic

2.3 Markov decision process

This section introduces Markov decision processes as models of action under uncertainty and explains their components and long-term policies.

Key terms: Markov chain, Markov decision process, prediction, optimization, state, action, transition probability, transition function, reward, policy

2.4 Examples of Markov decision processes

This section models four business situations as Markov decision processes and explains why state and reward design matter.

Key terms: state, action, reward, transitions, dynamic pricing, workforce scheduling, queue length, staffing level, preventive maintenance, operating condition

2.5 Build a Markov Decision Process for Production Planning

This section builds and simulates a production-planning Markov decision process, comparing heuristic policies by their profit over time.

Key terms: state, action, inventory level, machine setup, Markovian, Poisson distribution, lambda, Normal distribution, mean, standard deviation

2.6 Reward engineering and constraint handling strategies

This section explains reward as a scalar learning signal and the need to balance business objectives and constraints when designing it.

Key terms: reward, scalar signal, reward engineering, multi-objective, reward function, objectives, constraints

2.6.1 Design rewards to be stepwise, whenever possible

This section explains why and how to provide stepwise rewards for multi-step business problems instead of relying only on end-of-episode feedback.

Key terms: episodic problems, sparse reward, dense reward, stepwise reward, Key Performance Indicators, numerical rewards, stepwise rewards, stockout, approximations, intermediate feedback

2.6.2 Inject constraint information into the state

This section explains how including constraint information in an agent’s state supports context-aware decisions and reduces reliance on learning constraints through penalties.

Key terms: state representation, constraints, state injection, state, current temperature, item sensitivity, power budget

2.6.3 Handle soft constrains with stepwise penalties

This section explains how graded rewards and penalties let an agent handle soft constraints while balancing competing priorities.

Key terms: soft constraints, penalty shaping, gradual penalty, graded feedback, pickup_time, reward system, gradient of penalties

2.6.4 Use action masking with penalties to handle hard constraints

This section explains how action masking and gentle penalties can enforce hard constraints while encouraging safer feasible actions.

Key terms: hard constraints, action masking, state, feasibility, gentle penalties, weight limit, borderline cases

2.6.5 Avoid mismatched scales with reward normalization / balancing

This section explains how mismatched reward scales distort agent behavior and introduces ways to balance reward components and guide learning.

Key terms: scale imbalance, reward components, risk penalties, common unit, normalization, reward and penalty values, Lagrangian methods, constraints, rigid penalties, potential-based reward shaping

2.6.6 Avoid deceptive shortcuts in reward function

This section explains how proxy rewards can produce deceptive shortcuts and how better reward design and monitoring can discourage them.

Key terms: proxy, true objective, deceptive shortcut, click count, revenue, bounce rate, normalized combination, simulations, audits, scenario testing

2.7 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 1: Real-world Decision Making with Reinforcement Learning Chapter 3: Design Custom Environments for Reinforcement Learning Algorithms →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.