A transition dynamic describes how an environment changes after an agent takes an action. It connects the current state and action to the next state, including uncertainty where outcomes are not guaranteed. Realistic transitions make a simulation useful for business decisions. section 3.1 section 3.5 section 4.4
Parcel delivery
If an agent chooses to visit a customer, the environment updates its location, marks the package delivered, and advances time according to distance and speed. The transition logic also reflects consequences of ignoring traffic or time windows. section 3.1
In a custom environment, reset initializes the state. Each call to step applies the transition, calculates a reward, updates the state and time step, and checks whether the episode should end. The transition determines what happens; the reward signals how desirable the result is. section 3.1
Allocation under uncertainty
Suppose a project has a base success probability of 0.70. Each unit allocated beyond the first adds 0.05, up to a probability cap of 0.95. Allocating one unit gives a 0.70 success probability; allocating three gives 0.80. More investment improves the odds but leaves fewer resources for other projects. section 4.4
Listing 4.4 Transition Dynamics Function
def transition_probability(mdp: ResourceAllocationMDP,
state: tuple,
action: tuple,
next_state: tuple) -> float:
budget, proj_outcomes = state #A
next_budget, next_proj_outcomes = next_state #A
project_id, allocation = action #A
if project_id == -1: #B
return 1.0 if state == next_state else 0.0 #B
if allocation > budget or allocation <= 0: #C
return 0.0
if proj_outcomes[project_id] != 0: #D
return 0.0
if next_budget != budget - allocation: #E
return 0.0
base_prob = mdp.project_params[project_id]["success_prob"] #F
enhanced_prob = min(0.95, base_prob + (allocation - 1) * 0.05)#F
expected_outcomes = list(proj_outcomes)
if next_proj_outcomes[project_id] == 1: #G
expected_outcomes[project_id] = 1 #G
if tuple(expected_outcomes) == next_proj_outcomes: #G
return enhanced_prob #G
elif next_proj_outcomes[project_id] == -1: #H
expected_outcomes[project_id] = -1 #H
if tuple(expected_outcomes) == next_proj_outcomes: #H
return 1.0 - enhanced_prob #H
return 0.0 #I
Inspect a transition probability function that checks budget and project eligibility before calculating the enhanced success probability. section 4.4
Test transitions before training: random agents can reveal bugs, and visual debugging can help with spatial environments. Refine the simulation iteratively so its feedback and constraints remain clear. section 3.5
