liveBook · Manning

Transition dynamic

As explained in Applied Reinforcement Learning

The rules governing how an environment changes from one state to another after an action, including effects of randomness.

A transition dynamic describes how an environment changes after an agent takes an action. It connects the current state and action to the next state, including uncertainty where outcomes are not guaranteed. Realistic transitions make a simulation useful for business decisions. section 3.1 section 3.5 section 4.4

Parcel delivery

If an agent chooses to visit a customer, the environment updates its location, marks the package delivered, and advances time according to distance and speed. The transition logic also reflects consequences of ignoring traffic or time windows. section 3.1

In a custom environment, reset initializes the state. Each call to step applies the transition, calculates a reward, updates the state and time step, and checks whether the episode should end. The transition determines what happens; the reward signals how desirable the result is. section 3.1

Allocation under uncertainty

Suppose a project has a base success probability of 0.70. Each unit allocated beyond the first adds 0.05, up to a probability cap of 0.95. Allocating one unit gives a 0.70 success probability; allocating three gives 0.80. More investment improves the odds but leaves fewer resources for other projects. section 4.4

Listing 4.4 Transition Dynamics Function
def transition_probability(mdp: ResourceAllocationMDP, 
                         state: tuple, 
                         action: tuple, 
                         next_state: tuple) -> float:
 
    budget, proj_outcomes = state #A
    next_budget, next_proj_outcomes = next_state #A
    project_id, allocation = action #A
    
    if project_id == -1: #B
        return 1.0 if state == next_state else 0.0 #B
    
    if allocation > budget or allocation <= 0: #C
        return 0.0
    
    if proj_outcomes[project_id] != 0: #D
        return 0.0
    
    if next_budget != budget - allocation: #E
        return 0.0
     
    base_prob = mdp.project_params[project_id]["success_prob"] #F
    enhanced_prob = min(0.95, base_prob + (allocation - 1) * 0.05)#F
    
    expected_outcomes = list(proj_outcomes)
    
    if next_proj_outcomes[project_id] == 1: #G
        expected_outcomes[project_id] = 1 #G
        if tuple(expected_outcomes) == next_proj_outcomes: #G
            return enhanced_prob #G
    
    elif next_proj_outcomes[project_id] == -1: #H
        expected_outcomes[project_id] = -1 #H
        if tuple(expected_outcomes) == next_proj_outcomes: #H
            return 1.0 - enhanced_prob #H
    
    return 0.0 #I

Inspect a transition probability function that checks budget and project eligibility before calculating the enhanced success probability. section 4.4

Test transitions before training: random agents can reveal bugs, and visual debugging can help with spatial environments. Refine the simulation iteratively so its feedback and constraints remain clear. section 3.5

How Applied Reinforcement Learning covers it

How deep it goes
Defines transition logic and works through state updates, probability calculations, and validity checks in a transition function.
Examples
Uses parcel delivery updates and project allocation with success probabilities that increase with investment.
What you can do after
Build a transition probability function and test environment behavior with random agents.
How it connects
Connects state and action spaces to reward functions and policy evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.