Chapter 1: Real-world Decision Making with Reinforcement Learning
The introduction frames business decisions as repeated choices under uncertainty and connects that challenge to reinforcement learning.
After this chapter you will be able to
- Sequential decisions and reinforcement learning. Explain why business decision-making is sequential under uncertainty and state how the introduction relates it to reinforcement learning.
- Learning through interaction. Explain how an agent learns a policy by interacting with an environment, using AlphaStar as an example.
- Long-term rewards and credit. Explain why an agent explores actions for long-term reward and what credit assignment means.
- Machine learning approach comparison. Distinguish unsupervised, supervised and reinforcement learning by what guides their learning and what they learn to do.
- Sequential business decision making. Explain how reinforcement learning can address a business decision sequence, contrasting it with a prediction.
- Internal and external factors. Distinguish internal and external business factors by the degree of control a business has, including cases of partial control.
- Descriptive analysis and forecasting. Distinguish descriptive analysis of the past from forecasting future external factors, including the assumption behind forecasting in this section.
- Explanatory analysis and optimization. Distinguish explanatory analysis of past internal outcomes from optimization of future controllable decisions.
- Model based and data driven methods. Explain how model-based and data-driven approaches represent a problem and how they can be combined.
- Business optimization suitability. Explain how decision level, cycle, dimensions, and quantifiability help identify problems suited to business optimization.
- Optimization model inputs. Distinguish external factors supplied as parameters from actions represented by decision variables, and describe what sensitivity analysis reveals.
- Optimization objectives. Describe an objective function as what the model minimizes or maximizes, including the possibility of multiple objectives.
- Constraints and model usefulness. Explain how business constraints limit an optimization model and why their treatment must balance realism with solvability.
- Optimization model outputs. Describe the objective metrics and decision-variable values produced by a solved optimization model, using the vehicle-routing example.
- Optimization Problem Characteristics. Explain what makes the examples suitable for business optimization.
- Retail Inventory Replenishment. Describe the inputs, decisions, objective, and service-level constraint in retail inventory replenishment.
- Process Quality and Evaluation. Explain why model quality should be evaluated across different situations over time, using variance and bias.
- Bias Variance Trade-off. Explain the trade-off between bias and variance and who must decide how to balance it.
- Business and Modeling Constraints. Describe the two sources of constraints that must be respected when building a business optimization model.
- Operations research problem formulation. Explain how operations research turns a business problem into decision variables, an objective, and constraints, using the bakery example.
- Optimization model type distinctions. Distinguish linear programming, mixed integer programming, and nonlinear programming by their relationships and variable requirements.
- Simulation under business uncertainty. Explain how stochastic simulation explores a range of possible system behaviors under uncertainty, rather than directly finding a single best solution. Describe how a coffee shop could compare changes using performance measures.
- Discrete event system evolution. Explain how discrete event simulation tracks a system over time and why that helps with coffee shop decisions.
- Long-term strategic feedback. Explain how system dynamics helps managers anticipate long-term effects of business decisions, using a returns or hiring policy as an example.
- Causal feedback loop diagrams. Describe how causal loop diagrams show relationships between variables and distinguish reinforcing from balancing loops.
- Interdependent strategic business decisions. Explain how game theory models multiple players pursuing their own objectives, using a business example to illustrate how one player's best move depends on another's decision.
- Nash equilibrium and stability. Explain Nash equilibrium in terms of whether a player can improve its outcome by changing its decision while the other players' decisions stay the same.
- Agent environment learning loop. Trace the agent-environment interaction and explain how repeated feedback leads to a policy.
- Sequential and delayed consequences. Explain why reinforcement learning considers long-term outcomes rather than only immediate rewards.
- Adaptive business decision systems. Explain why learning through experience can help business decisions adapt when conditions change, and illustrate it with a section example.
- Reinforcement learning problem fit. Explain why reinforcement learning is not appropriate for every business decision problem.
- Adaptation to changing conditions. Explain why changing business conditions can undermine a classical optimization plan and how reinforcement learning learns without knowing everything upfront.
- Complementary Optimization Approaches. Explain how reinforcement learning extends rather than replaces classical optimization.
- Reinforcement Learning Practical Limitations. Describe the practical costs and risks of reinforcement learning, including its interaction needs and difficulty of explanation.
Section by section
Introduction
The introduction frames business decisions as repeated choices under uncertainty and connects that challenge to reinforcement learning.
- Adaptation under resource limits: Explain why businesses must adapt to uncertainty while recognizing what they can control and the resources available to them.
Why it matters: This frames why business decisions must account for both resource limits and an uncertain environment. - Sequential decisions and reinforcement learning: Explain why business decision-making is sequential under uncertainty and state how the introduction relates it to reinforcement learning.
Why it matters: This connects the book's reinforcement learning topic to the repeated decisions businesses face.
Key terms: limited resources, uncertainties, strengths and weaknesses, opportunities and threats, sequential decisions, uncertainty, reinforcement learning
1.1 What reinforcement learning really enables?
This section introduces reinforcement learning as learning actions from experience and contrasts it with other machine learning approaches for business decisions.
- Learning through interaction: Explain how an agent learns a policy by interacting with an environment, using AlphaStar as an example.
Why it matters: This is the basic learning pattern behind the methods in the book. - Long-term rewards and credit: Explain why an agent explores actions for long-term reward and what credit assignment means.
Why it matters: Good reinforcement learning decisions depend on consequences over time, not just an immediate result. - Machine learning approach comparison: Distinguish unsupervised, supervised and reinforcement learning by what guides their learning and what they learn to do.
Why it matters: The comparison clarifies why predicting an outcome and learning what action to take are different tasks. - Sequential business decision making: Explain how reinforcement learning can address a business decision sequence, contrasting it with a prediction.
Why it matters: This connects reinforcement learning to the business optimization problems explored in the book.
Key terms: agent, environment, policies, AlphaStar, explore, cumulative outcomes, long-term value, credit assignment, rewards, penalties
1.2 Different types of business analysis
This section distinguishes business analysis questions by time frame and degree of control, and contrasts model-based and data-driven ways to answer them.
- Internal and external factors: Distinguish internal and external business factors by the degree of control a business has, including cases of partial control.
Why it matters: The degree of control helps you choose which questions to ask about a business situation. - Descriptive analysis and forecasting: Distinguish descriptive analysis of the past from forecasting future external factors, including the assumption behind forecasting in this section.
Why it matters: Separating a historical description from a forecast clarifies what an analysis can answer. - Explanatory analysis and optimization: Distinguish explanatory analysis of past internal outcomes from optimization of future controllable decisions.
Why it matters: Understanding why a result occurred is different from choosing what to do next. - Model based and data driven methods: Explain how model-based and data-driven approaches represent a problem and how they can be combined.
Why it matters: These approaches offer different ways to analyze and influence future business outcomes. - Analytical tool selection: Explain why the business question and the chosen approach both matter, and state the section’s qualification about reinforcement learning.
Why it matters: A well-phrased question still needs a suitable method to produce a useful answer.
Key terms: external factors, internal factors, partial control, Descriptive Analysis, Prediction, Forecasting, Explanatory Analysis, Optimization, Model-based, Data-driven
1.3 Business optimization definition
This section outlines when business optimization is typically useful and defines the inputs, objectives, constraints, and outputs of an optimization model.
- Business optimization suitability: Explain how decision level, cycle, dimensions, and quantifiability help identify problems suited to business optimization.
Why it matters: These characteristics help you recognize when an optimization model is a reasonable fit. - Optimization model inputs: Distinguish external factors supplied as parameters from actions represented by decision variables, and describe what sensitivity analysis reveals.
Why it matters: Separating external conditions from choices clarifies what the model takes as input and what it must decide. - Optimization objectives: Describe an objective function as what the model minimizes or maximizes, including the possibility of multiple objectives.
Why it matters: The objective specifies what a model is trying to achieve. - Constraints and model usefulness: Explain how business constraints limit an optimization model and why their treatment must balance realism with solvability.
Why it matters: A model that ignores important limitations or is too complicated to solve may not be useful. - Optimization model outputs: Describe the objective metrics and decision-variable values produced by a solved optimization model, using the vehicle-routing example.
Why it matters: Both the achieved outcome and the recommended actions are needed to use a model's solution.
Key terms: decision level, decision cycle, decision dimensions, decision quantifiability, external factors, parameters, actions, decision variables, sensitivity analysis, objective
1.4 Examples of business optimization problems
This section illustrates recurring business optimization problems and their shared structure through operational examples.
- Optimization Problem Characteristics: Explain what makes the examples suitable for business optimization.
Why it matters: Recognizing these characteristics helps identify decisions that may benefit from optimization. - Retail Inventory Replenishment: Describe the inputs, decisions, objective, and service-level constraint in retail inventory replenishment.
Why it matters: This example shows how inputs, decisions, costs, and a constraint combine in an operational optimization problem. - Delivery and Bike Routing: Describe the routing decisions and objectives in delivery fleets and bike-sharing rebalancing.
Why it matters: The two examples show how routing decisions serve different operational objectives. - Production Scheduling and Capacity: Describe production scheduling decisions and explain the issue of oversubscribed tasks.
Why it matters: Production scheduling illustrates how limited capacity forces operational priorities. - Workforce Shift Scheduling: Describe the decisions, inputs, and objectives in workforce shift scheduling.
Why it matters: Workforce scheduling shows how operational objectives must coexist with staffing and legal requirements. - Perishable Goods Dynamic Pricing: Describe how operational dynamic pricing is framed for goods that lose value over time.
Why it matters: Dynamic pricing illustrates a measurable, recurring pricing decision for perishable inventory.
Key terms: business optimization, operational, recurring, quantifiable, inventory replenishment, demand forecasts, supplier lead times, order quantities, service level agreement, vehicle routing
1.5 Challenges in business optimization problems
This section explains how bias, variance, and practical constraints affect the quality of business optimization models.
- Process Quality and Evaluation: Explain why model quality should be evaluated across different situations over time, using variance and bias.
Why it matters: A model should perform consistently near its target, not just produce one good result. - Bias Variance Trade-off: Explain the trade-off between bias and variance and who must decide how to balance it.
Why it matters: Improving one aspect of model quality may come at the expense of another. - Business and Modeling Constraints: Describe the two sources of constraints that must be respected when building a business optimization model.
Why it matters: A model must fit both the business situation and the limits of the method used to represent it.
Key terms: variance, bias, process quality, trade-off, constraints, modeling approach
1.6 Classical business optimization models
This section introduces classical modeling methods as longstanding foundations of business decision-making systems.
1.6.1 Operations research
This section introduces operations research through a bakery optimization problem, distinguishes three model types, and describes the role of solvers.
- Operations research problem formulation: Explain how operations research turns a business problem into decision variables, an objective, and constraints, using the bakery example.
Why it matters: Formulation makes the choices, goal, and limits explicit so they can be optimized. - Optimization model type distinctions: Distinguish linear programming, mixed integer programming, and nonlinear programming by their relationships and variable requirements.
Why it matters: These distinctions help identify the kind of mathematical model a business problem requires. - Optimization solver role: Describe what a solver does after an operations research problem has been formulated.
Why it matters: A formulation can be handed to a solver to find an optimized result.
Key terms: operations research, decision variables, objective function, constraints, linear programming, LP, mixed integer programming, MIP, nonlinear programming, NLP
1.6.2 Stochastic simulation
This section explains how stochastic and discrete event simulations explore business systems under uncertainty, using a coffee shop as an example.
- Simulation under business uncertainty: Explain how stochastic simulation explores a range of possible system behaviors under uncertainty, rather than directly finding a single best solution. Describe how a coffee shop could compare changes using performance measures.
Why it matters: Simulation helps you assess possible consequences of a business change when arrivals, service times, or other conditions vary. - Queues and changing processes: Distinguish queueing theory from stochastic processes using the coffee shop examples.
Why it matters: These approaches focus attention on different aspects of a system: lines and delays versus changes over time. - Randomized scenarios and data: Describe the roles of Monte Carlo simulation and statistical inference modeling in stochastic simulation.
Why it matters: Randomized scenarios and data-fitted distributions help simulations represent uncertainty and estimate its consequences. - Discrete event system evolution: Explain how discrete event simulation tracks a system over time and why that helps with coffee shop decisions.
Why it matters: Tracking timed events gives a clearer picture of customer flow and resource use when evaluating operational changes.
Key terms: stochastic simulation, uncertainty, key performance indicators, queueing theory, stochastic processes, Monte Carlo simulation, statistical inference modeling, probability distributions, risk exposure, discrete event simulation
1.6.3 System dynamics
This section explains how system dynamics models feedback and changes over time to support long-term business decisions.
- Long-term strategic feedback: Explain how system dynamics helps managers anticipate long-term effects of business decisions, using a returns or hiring policy as an example.
Why it matters: This helps explain why a decision that looks beneficial now can cause problems later. - Causal feedback loop diagrams: Describe how causal loop diagrams show relationships between variables and distinguish reinforcing from balancing loops.
Why it matters: Recognizing the two kinds of feedback helps you see whether interactions amplify or resist change. - Stock and flow simulation: Describe the roles of stocks and flows in simulating system behavior over time.
Why it matters: Stocks and flows provide a way to represent how quantities build up and change during a simulation.
Key terms: system dynamics, feedback, time-delayed relationships, causal loop diagram, reinforcing loops, balancing loops, stock-and-flow diagrams, stocks, flows, differential equations
1.6.4 Game theory
This section explains how game theory models interdependent business decisions and defines Nash equilibrium as a stable balance of strategies.
- Interdependent strategic business decisions: Explain how game theory models multiple players pursuing their own objectives, using a business example to illustrate how one player's best move depends on another's decision.
Why it matters: This helps you recognize business decisions that cannot be optimized as if competitors acted in isolation. - Nash equilibrium and stability: Explain Nash equilibrium in terms of whether a player can improve its outcome by changing its decision while the other players' decisions stay the same.
Why it matters: This gives you a precise way to describe a stable set of strategic decisions.
Key terms: Game Theory, players, strategic decision-making, objectives, Nash Equilibrium, strategies, unilaterally
1.7 Reinforcement learning for business optimization
This section explains how reinforcement learning learns adaptive business decisions through interaction, long-term feedback, and trial and error, while recognizing when simpler approaches may be preferable.
- Agent environment learning loop: Trace the agent-environment interaction and explain how repeated feedback leads to a policy.
Why it matters: This loop is the basic way an RL system learns to make decisions from experience. - Sequential and delayed consequences: Explain why reinforcement learning considers long-term outcomes rather than only immediate rewards.
Why it matters: A decision that looks good immediately may affect outcomes later. - Adaptive business decision systems: Explain why learning through experience can help business decisions adapt when conditions change, and illustrate it with a section example.
Why it matters: Changing markets can make a fixed decision rule less useful over time. - Learning versus other agents: Distinguish an RL agent from other kinds of agents by the way it improves its decisions.
Why it matters: The word agent alone does not imply that a system learns through reinforcement. - Reinforcement learning problem fit: Explain why reinforcement learning is not appropriate for every business decision problem.
Why it matters: Choosing a suitable tool matters more than applying RL everywhere.
Key terms: agent, environment, state, action, reward, policy, long-term planning, sequential decisions, delayed consequences, short-term rewards
1.8 Limitations in classical models and reinforcement learning
This section compares classical optimization with reinforcement learning, explaining their complementary strengths and practical limitations.
- Adaptation to changing conditions: Explain why changing business conditions can undermine a classical optimization plan and how reinforcement learning learns without knowing everything upfront.
Why it matters: This contrast explains why learning from interaction can be useful when business conditions change. - Complementary Optimization Approaches: Explain how reinforcement learning extends rather than replaces classical optimization.
Why it matters: You can carry classical optimization thinking into reinforcement learning rather than discard it. - Reinforcement Learning Practical Limitations: Describe the practical costs and risks of reinforcement learning, including its interaction needs and difficulty of explanation.
Why it matters: These limitations affect whether and how an RL solution can be developed and trusted in a business.
Key terms: classical models, constraints, reinforcement learning, feedback, adaptability, decision variables, objective functions, trade-offs, interactions, simulations
1.9 Summary
Where this is used later
- State: Anatomy of sequential decision making (chapter 2) · Develops the sequential decision-making context.
- Tradeoff between exploitation and exploration (chapter 5) · Develops the role of exploration in learning which decisions work.
- Design rewards to be stepwise, whenever possible (chapter 2) · Considers how rewards can be designed for sequential decisions.
- Build a Markov Decision Process for Production Planning (chapter 2) · Building a production-planning model requires identifying relevant conditions and decisions.
- Reward engineering and constraint handling strategies (chapter 2) · Reward design builds on identifying what the decision process should achieve.
- Solving capacitated vehicle routing problem with Monte Carlo tree search (chapter 7) · Applies a search method to a vehicle routing problem.
- Perishable product dynamic pricing environment (chapter 3) · Develops an environment for perishable product dynamic pricing.
- Designing the evaluation (chapter 12) · Evaluation assesses how a system performs rather than relying on a single outcome.
Concepts explained in this chapter
- business optimization problemBusiness optimization is the application of analytical and mathematical methods to improve decisions about business operations and resource allocation.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns to choose actions through interactions with an environment and feedback about their consequences.
- decision variableA decision variable is a quantity whose value is chosen by an optimization model to represent an available decision.
- objective functionA mathematical quantity that a learning or optimization procedure seeks to minimize or maximize.
- constraintA restriction that limits the values or combinations allowed in an optimization problem, including which actions are valid in a state.
- AI agentAn entity in reinforcement learning that observes an environment, selects actions, and improves its behavior from feedback.
- environmentAn environment is the external system or setting with which an agent interacts and whose responses produce consequences for the agent's actions.
- policyA policy is a decision rule, possibly probabilistic, that assigns actions or action probabilities to states for an agent.
- reward signalA reward is a numerical feedback signal indicating the desirability of an outcome or transition after an agent's action.
- optimizationOptimization is the selection of controllable decisions that maximize or minimize an objective subject to constraints.
- sequential decision-makingA decision process in which actions are chosen over multiple steps while relevant states, outcomes, or future conditions are not fully known.
- model-based approachA model-based method solves a decision problem by using a known or learned model of the environment to plan ahead.
- actionAn action is a choice made by an agent that can affect the subsequent state of its environment.
- agent-environment interaction loopThe recurring reinforcement-learning cycle in which an agent observes a state, takes an action, receives feedback, and reaches a new state.
- stateA state describes the relevant condition of an environment at a particular point in a sequential decision process.
- operations researchOperations research is the discipline of applying mathematical models, logic, and optimization methods to support decision-making under constraints.
- bike-sharing station rebalancingBike-sharing station rebalancing is the logistical redistribution of bikes among stations to reduce shortages and surpluses.
- dynamic pricingDynamic pricing is the adjustment of prices over time in response to changing demand, inventory conditions, expiration, or competitive factors.
- inventory replenishmentInventory replenishment is the process of determining when and how much product to reorder for stocked locations.
- production schedulingProduction scheduling is the assignment and sequencing of manufacturing tasks over available machines and time periods.
- vehicle routing problemThe vehicle routing problem is an optimization problem that determines vehicle routes for serving locations, typically while minimizing travel-related cost or distance.
- workforce shift schedulingWorkforce scheduling is the assignment of staff levels and working periods to meet anticipated demand while balancing labor cost and service requirements.
- classical optimization methodA classical optimization method is a mathematical procedure for solving a decision problem defined by objectives and constraints.
- credit assignmentCredit assignment is the problem of determining which earlier decisions contributed to a later success or failure.
- Monte Carlo simulationMonte Carlo estimation approximates a quantity by averaging values obtained from random samples.
- discrete event simulationDiscrete event simulation models a system as an ordered sequence of instantaneous events that change its state over time.
- stochastic simulationA stochastic simulation is a computational model that represents randomness and explores the range of possible system behaviors under uncertainty.
- bias-variance tradeoffThe bias-variance tradeoff is the balance between systematic estimation error and variability across samples or outcomes.
- long-term planningPlanning that evaluates how present actions affect outcomes over an extended sequence of future decisions.
- service-level agreementA service level agreement is a formal commitment specifying expected service performance, such as the availability or fulfillment level of a product or service.
- supplier lead timeSupplier lead time is the elapsed time between placing an order with a supplier and receiving it.
- decision cycleA classification of a decision by whether it recurs periodically, occurs once, or occurs occasionally.
- decision dimensionA classification indicating whether a decision concerns one entity or multiple entities.
- decision levelA classification of a decision by its strategic, tactical, or operational scope.
- decision quantifiabilityThe degree to which a decision can be represented and evaluated quantitatively rather than qualitatively.
- sensitivity analysisAn analysis of how changes in model inputs or parameters affect model outputs or objectives.
- supervised learningA machine-learning technique that learns mappings from inputs to target outputs using labeled examples.
- probability distributionA probability distribution specifies the probabilities assigned to the possible values of a random variable.
- simulationSimulation is the execution of a model that reproduces interactions or outcomes of an environment without operating directly in the real environment.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
