A coffee shop cannot predict every morning rush, but it can simulate many possible ones before deciding whether to add staff. Monte Carlo simulation estimates a quantity by averaging outcomes from random samples. Running hundreds or thousands of scenarios can help estimate performance and risk under uncertain conditions. section 1.6.2 section 9.3
Testing coffee shop decisions
Simulate customer traffic under choices such as hiring another barista, adding a self-serve kiosk, or changing staffing by time of day. Compare measures such as average wait time, queue length, and lost sales across randomized busy and quiet periods. section 1.6.2
In reinforcement learning, Monte Carlo methods can learn from complete episodes rather than updating after each step. For a seasonal promotion, that means waiting until the campaign ends, then using its overall outcome to adjust the strategy. Temporal-difference learning instead updates incrementally using observed rewards and predicted future value, a process called bootstrapping. section 4.1 section 6.7
REINFORCE uses sampled trajectories to estimate a policy gradient: a direction for making rewarding actions more likely. For each action, it uses the return-to-go—the discounted sum of rewards from that step onward. Excluding earlier rewards avoids assigning the action credit for events it could not have caused and reduces noise. section 9.3
The estimate averages contributions from N sampled trajectories. Each contribution combines the policy’s log-probability gradient for an action with that action’s return-to-go. section 9.3
The REINFORCE gradient estimate is unbiased, but an individual batch can be far from the true gradient when sampled returns are unusually good or bad. More trajectories can improve the estimate; baselines can reduce variance. Episode-based learning can also be slow when outcomes take a long time to arrive. section 9.3 section 4.1










