liveBook · Manning

Monte Carlo simulation

As explained in Applied Reinforcement Learning

Monte Carlo estimation approximates a quantity by averaging values obtained from random samples.

A coffee shop cannot predict every morning rush, but it can simulate many possible ones before deciding whether to add staff. Monte Carlo simulation estimates a quantity by averaging outcomes from random samples. Running hundreds or thousands of scenarios can help estimate performance and risk under uncertain conditions. section 1.6.2 section 9.3

Testing coffee shop decisions

Simulate customer traffic under choices such as hiring another barista, adding a self-serve kiosk, or changing staffing by time of day. Compare measures such as average wait time, queue length, and lost sales across randomized busy and quiet periods. section 1.6.2

In reinforcement learning, Monte Carlo methods can learn from complete episodes rather than updating after each step. For a seasonal promotion, that means waiting until the campaign ends, then using its overall outcome to adjust the strategy. Temporal-difference learning instead updates incrementally using observed rewards and predicted future value, a process called bootstrapping. section 4.1 section 6.7

REINFORCE uses sampled trajectories to estimate a policy gradient: a direction for making rewarding actions more likely. For each action, it uses the return-to-go—the discounted sum of rewards from that step onward. Excluding earlier rewards avoids assigning the action credit for events it could not have caused and reduces noise. section 9.3

∇ θJ(θ) ≈ (1/N) Σᵢ₌₁ᴺ [ Σₜ ∇ θ log π θ(aₜⁱ|sₜⁱ) Gₜⁱ

The estimate averages contributions from N sampled trajectories. Each contribution combines the policy’s log-probability gradient for an action with that action’s return-to-go. section 9.3

The REINFORCE gradient estimate is unbiased, but an individual batch can be far from the true gradient when sampled returns are unusually good or bad. More trajectories can improve the estimate; baselines can reduce variance. Episode-based learning can also be slow when outcomes take a long time to arrive. section 9.3 section 4.1

How Applied Reinforcement Learning covers it

How deep it goes
Works through a sample-based policy gradient, return-to-go, and variance; contrasts complete-episode learning with incremental updates.
Examples
Uses coffee shop staffing scenarios and a seasonal promotion campaign to illustrate simulation and complete-episode learning.
What you can do after
Evaluate uncertain decisions and understand how sampled trajectories estimate a policy gradient.
How it connects
Connects Monte Carlo estimation to REINFORCE, credit assignment, temporal-difference learning, and bootstrapping.

Also explained in

Compare how 11 books explain Monte Carlo simulation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.