liveBook · Manning

What you will learn in Applied Reinforcement Learning

Business optimization and LLM fine-tuning

by Hadi Aghazadeh · Manning Early Access Program (MEAP)

Chapter by chapter

Chapter 1: Real-world Decision Making with Reinforcement Learning

The introduction frames business decisions as repeated choices under uncertainty and connects that challenge to reinforcement learning.

1.1 What reinforcement learning really enables? · 1.2 Different types of business analysis · 1.3 Business optimization definition · 1.4 Examples of business optimization problems · 1.5 Challenges in business optimization problems · 1.6 Classical business optimization models · 1.6.1 Operations research · 1.6.2 Stochastic simulation · 1.6.3 System dynamics · 1.6.4 Game theory · 1.7 Reinforcement learning for business optimization · 1.8 Limitations in classical models and reinforcement learning · 1.9 Summary

What you learn in chapter 1 →

Chapter 2: Markov Decision Process: Turning Problems intro Solvable Models

This introduction frames Markov decision processes as a way to break down and reconnect complex business decisions over time.

2.1 State: Anatomy of sequential decision making · 2.2 Markov chain and Markov property · 2.3 Markov decision process · 2.4 Examples of Markov decision processes · 2.5 Build a Markov Decision Process for Production Planning · 2.6 Reward engineering and constraint handling strategies · 2.6.1 Design rewards to be stepwise, whenever possible · 2.6.2 Inject constraint information into the state · 2.6.3 Handle soft constrains with stepwise penalties · 2.6.4 Use action masking with penalties to handle hard constraints · 2.6.5 Avoid mismatched scales with reward normalization / balancing · 2.6.6 Avoid deceptive shortcuts in reward function · 2.7 Summary

What you learn in chapter 2 →

Chapter 3: Design Custom Environments for Reinforcement Learning Algorithms

This introduction explains why reinforcement learning agents need simulated environments for safe trial-and-error learning in business settings.

3.1 Conceptual framework for designing business environment · 3.2 Warehouse order picking environment · 3.3 Perishable product dynamic pricing environment · 3.4 Trailer loading and packing environment · 3.5 Summary

What you learn in chapter 3 →

Chapter 4: Perfect Knowledge, Optimal Policy: Dynamic Programming

This introduction contrasts model-based and model-free methods and situates dynamic programming as a recursive approach to solving Markov decision processes with a known model.

4.1 Paradigms on solving Markov decision process · 4.2 The domino decision rule: Bellman equations · 4.3 Solving bellman equations: Generalized Policy Iteration · 4.4 Hands-on code: solving a resource allocation problem · 4.5 Limitations of dynamic programming · 4.6 Summary

What you learn in chapter 4 →

Chapter 5: Contextual Bandit: Optimizing Stochastic One-step Decisions

This introduction motivates bandits as experience-based decision methods and distinguishes pure from contextual bandits.

5.1 Bandits as lightweight reinforcement learning · 5.2 Tradeoff between exploitation and exploration · 5.3 Simulating an Ad campaign problem with bandit algorithms · 5.4 Quantifying bandit algorithms performance with Regret · 5.5 Dynamic personalized discounting with contextual bandits · 5.6 Beyond stationary bandit problems · 5.7 Summary

What you learn in chapter 5 →

Chapter 6: Tabular Reinforcement Learning

The introduction positions temporal-difference learning as a foundation of reinforcement learning and briefly contrasts tabular and approximation-based methods.

6.1 Temporal difference learning · 6.2 A concrete example: restaurant table scheduling · 6.3 Off policy vs on policy learning · 6.4 Tabular Reinforcement Learning: Q-learning and SARSA · 6.4.1 SARSA: learning from what you actually do · 6.4.2 Q-learning: learning from what you should do · 6.5 TD(λ) and Eligibility traces · 6.6 Gas station fuel purchase scheduling with tabular methods · 6.7 Summary

What you learn in chapter 6 →

Chapter 7: Monte Carlo Tree Search: Searching with Reinforcement Learning Principles

This introduction presents Monte Carlo tree search as a blend of simulation, planning, and learning for exploring sequential decisions under uncertainty.

7.1 Tree data structure and its applications in business · 7.2 Fundamental tree search algorithms · 7.3 Monte Carlo tree search theory and algorithm · 7.4 Solving capacitated vehicle routing problem with Monte Carlo tree search · 7.5 Summary

What you learn in chapter 7 →

Chapter 8: Deep Q-Networks for High-dimensional Data

This introduction uses radiotherapy planning to motivate moving beyond tabular reinforcement learning for complex, high-dimensional decisions.

8.1 From tabular to linear function approximation · 8.2 Deep learning fundamentals · 8.3 DQN theory: Q-learning meets neural networks · 8.4 Implementation of DQN for treatment optimization · 8.5 DQN beyond treatment: when and how to use DQN · 8.6 Summary

What you learn in chapter 8 →

Chapter 9: The Calculus of Decisions: Policy Gradient Methods

This introduction motivates direct policy optimization using the challenges of continuous, coupled data center cooling control.

9.1 Why optimizing the policy directly? · 9.2 The mathematics of policy gradients · 9.3 The REINFORCE algorithm · 9.4 Variance reduction through baselines · 9.5 Attention mechanism: learning what to look at · 9.6 The Transformer architecture · 9.7 Case study: data center cooling optimization · 9.8 Summary

What you learn in chapter 9 →

Chapter 10: Fine-tuning large language models with PPO

The introduction explains why fluent medical chatbot responses may be unsafe and how reinforcement learning can use human judgments to improve alignment.

10.1 A New Frontier for Reinforcement Learning: Language · 10.2 Overview of Large Language Models Architecture · 10.3 Mapping Language Model to a Markov Decision Process · 10.4 The RLHF Pipeline: Three Stages to Alignment · 10.5 Supervised Fine-Tuning (SFT) · 10.6 Learning a Reward Model from Human Preferences · 10.7 Proximal Policy Optimization: From REINFORCE to PPO · 10.8 Build a Medical Patient Support Chatbot with RLHF · 10.8.1 Pipeline configuration · 10.8.2 Safety signals · 10.8.3 Stage 1: Supervised fine-tuning · 10.8.4 Stage 2: reward model training · 10.8.5 Stage 3: PPO fine-tuning · 10.8.6 Evaluation · 10.9 Summary

What you learn in chapter 10 →

Chapter 11: Teach LLMs to reason with Verifiable Feedback using GRPO

This introduction explains when a programmatic verifier can replace a learned reward model for language-model tasks.

11.1 Teaching language models to reason on financial reports · 11.2 Reinforcement Learning with Verifiable Rewards · 11.3 LoRA: fine-tuning large models with tiny adapters · 11.4 GRPO: group relative policy optimization · 11.5 Training stability in GRPO · 11.5.1 Memory for logit computation · 11.5.2 Computing the reference policy without a second copy · 11.5.3 Gradient checkpointing · 11.5.4 Diagnosing training stability with KL divergence · 11.6 Five lessons from training GRPO at scale · 11.6.1 Lesson 1: Learning rate · 11.6.2 Lesson 2: Too hot RL vs too cold RL · 11.6.3 Lesson 3: KL penalty · 11.6.4 Lesson 4: Base model impact · 11.6.5 Lesson 5: KL signal is the key · 11.7 Building the financial reasoning agent · 11.7.1 Pipeline configuration · 11.7.2 LoRA injection · 11.7.3 FinQA data pipeline and prompt formatting · 11.7.4 RLVR verifier: extracting and scoring answers · 11.7.5 GRPO training: rollouts, advantages, and the update step · 11.7.6 Evaluation: comparing base and fine-tuned models · 11.8 Policy-based methods beyond language models · 11.8.1 Robotics and physical control · 11.8.2 Recommendation systems at scale · 11.8.3 Drug discovery and molecular design · 11.8.4 Code generation with execution feedback · 11.8.5 Operations research and combinatorial optimization · 11.8.6 Clinical decision support and treatment sequencing · 11.9 Summary

What you learn in chapter 11 →

Chapter 12: Building Reinforcement Learning systems: LLM-guided RL agent for portfolio management

This introduction frames practical RL as a systems-engineering challenge and illustrates it with an LLM-guided portfolio manager.

12.1 The common structure of deep reinforcement learning algorithms · 12.2 The gap between an algorithm and a system · 12.3 The capstone: an LLM-guided RL portfolio manager · 12.4 Framing and engineering the environment · 12.5 Representation engineering with a language model · 12.6 Wiring the model into the loop · 12.7 Designing the evaluation · 12.8 Results: does a better representation build a better system? · 12.9 Design principles for applied RL systems · 12.9.1 Representation has the greatest impact · 12.9.2 The reward is a specification the agent will exploit · 12.9.3 Guard the boundary of time: never let the future leak in · 12.9.4 The agent matters less than you think · 12.9.5 Evaluate as if you are trying to prove yourself wrong · 12.10 The pattern beyond trading · 12.11 Summary

What you learn in chapter 12 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.