liveBook · Manning

Simulation

As explained in Applied Reinforcement Learning

Simulation is the execution of a model that reproduces interactions or outcomes of an environment without operating directly in the real environment.

Simulation executes a model to reproduce an environment’s interactions or outcomes without acting in the real setting. For reinforcement learning, it provides a controlled sandbox where agents can explore, fail safely, and learn from feedback. section 3.0

Estimating customer transitions

Start a simulated software-service customer at “Buy Pro,” follow transition probabilities, and count the next states. Normalize the counts into probabilities, then compare estimates that include different previous-state histories. section 2.2

Listing 2.1 Simulate Markov Chain
def simulate_markov(P, initial_state, n_steps):
    n_states = P.shape[0]    #A
    chain = np.empty(n_steps, dtype=int)    #A
    chain[0] = initial_state    #B
    for t in range(1, n_steps):
        chain[t] = np.random.choice(n_states, p=P[chain[t-1]])    #C
    return chain

See a Markov-chain simulation that samples each next state from the transition probabilities for the current state. section 2.2

How Applied Reinforcement Learning covers it

How deep it goes
Demonstrates Markov-chain simulation and frequency-based estimates of next-state probabilities, alongside a high-level account of safe learning environments.
Examples
Uses software-service customer transitions for probability estimates and names warehouse robots, dynamic pricing, and trailer loading as environment scenarios.
What you can do after
Evaluate next-state probabilities by simulating customer transitions and normalizing observed state frequencies.
How it connects
Links simulation to reinforcement learning, Markov chains, optimization, operations research, and system dynamics.

Also explained in

Compare how 11 books explain simulation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.