A customer who installs an app might upgrade, cancel, or remain a user. A transition probability describes how likely a move from one state to another is, giving a model room for uncertainty rather than assuming every customer follows the same path. section 2.1
In a Markov decision process, the transition function P(s'|s,a) gives the probability of reaching next state s' from current state s after action a. The action affects how the system evolves, but does not necessarily determine the outcome. Rewards separately describe the immediate feedback from that outcome. section 2.3 section 4.4
Allocating resources to a project
Suppose a project has a base success probability of 0.70. The model adds 0.05 for each unit allocated beyond the first, up to a probability of 0.95. Allocating three units therefore gives a success probability of 0.80 and a failure probability of 0.20. The allocation also reduces the remaining budget, so success and failure lead to different next states. section 4.4
Listing 4.4 Transition Dynamics Function
def transition_probability(mdp: ResourceAllocationMDP,
state: tuple,
action: tuple,
next_state: tuple) -> float:
budget, proj_outcomes = state #A
next_budget, next_proj_outcomes = next_state #A
project_id, allocation = action #A
if project_id == -1: #B
return 1.0 if state == next_state else 0.0 #B
if allocation > budget or allocation <= 0: #C
return 0.0
if proj_outcomes[project_id] != 0: #D
return 0.0
if next_budget != budget - allocation: #E
return 0.0
base_prob = mdp.project_params[project_id]["success_prob"] #F
enhanced_prob = min(0.95, base_prob + (allocation - 1) * 0.05)#F
expected_outcomes = list(proj_outcomes)
if next_proj_outcomes[project_id] == 1: #G
expected_outcomes[project_id] = 1 #G
if tuple(expected_outcomes) == next_proj_outcomes: #G
return enhanced_prob #G
elif next_proj_outcomes[project_id] == -1: #H
expected_outcomes[project_id] = -1 #H
if tuple(expected_outcomes) == next_proj_outcomes: #H
return 1.0 - enhanced_prob #H
return 0.0 #I
The transition function checks the action and next-state budget before calculating the probability of a project outcome. section 4.4
Production planning with uncertain demand
A factory state tracks the day, scooter and bike inventories, and machine setup. A production action adds inventory, while randomly sampled demand reduces it. The combination determines the next state; a Poisson distribution can model discrete demand. section 2.5
Transition probabilities can be estimated from data for individual users or a broader population, but behavior and context can change. Some environments instead have deterministic transitions: in language generation, appending a chosen token determines the next state, while randomness can come from the policy that samples the token. section 2.1 section 10.3






