liveBook · Manning

Transition probability

As explained in Applied Reinforcement Learning

A transition probability is the probability of moving from one state to another in a stochastic system.

A customer who installs an app might upgrade, cancel, or remain a user. A transition probability describes how likely a move from one state to another is, giving a model room for uncertainty rather than assuming every customer follows the same path. section 2.1

In a Markov decision process, the transition function P(s'|s,a) gives the probability of reaching next state s' from current state s after action a. The action affects how the system evolves, but does not necessarily determine the outcome. Rewards separately describe the immediate feedback from that outcome. section 2.3 section 4.4

Allocating resources to a project

Suppose a project has a base success probability of 0.70. The model adds 0.05 for each unit allocated beyond the first, up to a probability of 0.95. Allocating three units therefore gives a success probability of 0.80 and a failure probability of 0.20. The allocation also reduces the remaining budget, so success and failure lead to different next states. section 4.4

Listing 4.4 Transition Dynamics Function
def transition_probability(mdp: ResourceAllocationMDP, 
                         state: tuple, 
                         action: tuple, 
                         next_state: tuple) -> float:
 
    budget, proj_outcomes = state #A
    next_budget, next_proj_outcomes = next_state #A
    project_id, allocation = action #A
    
    if project_id == -1: #B
        return 1.0 if state == next_state else 0.0 #B
    
    if allocation > budget or allocation <= 0: #C
        return 0.0
    
    if proj_outcomes[project_id] != 0: #D
        return 0.0
    
    if next_budget != budget - allocation: #E
        return 0.0
     
    base_prob = mdp.project_params[project_id]["success_prob"] #F
    enhanced_prob = min(0.95, base_prob + (allocation - 1) * 0.05)#F
    
    expected_outcomes = list(proj_outcomes)
    
    if next_proj_outcomes[project_id] == 1: #G
        expected_outcomes[project_id] = 1 #G
        if tuple(expected_outcomes) == next_proj_outcomes: #G
            return enhanced_prob #G
    
    elif next_proj_outcomes[project_id] == -1: #H
        expected_outcomes[project_id] = -1 #H
        if tuple(expected_outcomes) == next_proj_outcomes: #H
            return 1.0 - enhanced_prob #H
    
    return 0.0 #I

The transition function checks the action and next-state budget before calculating the probability of a project outcome. section 4.4

Production planning with uncertain demand

A factory state tracks the day, scooter and bike inventories, and machine setup. A production action adds inventory, while randomly sampled demand reduces it. The combination determines the next state; a Poisson distribution can model discrete demand. section 2.5

Transition probabilities can be estimated from data for individual users or a broader population, but behavior and context can change. Some environments instead have deterministic transitions: in language generation, appending a chosen token determines the next state, while randomness can come from the policy that samples the token. section 2.1 section 10.3

How Applied Reinforcement Learning covers it

How deep it goes
Defines conditional transition probabilities and implements them in resource allocation and stochastic production simulations.
Examples
Uses customer subscription states, factory demand, and project investments to illustrate transitions.
What you can do after
Builds a transition function from scratch for a resource-allocation model with action constraints and uncertain project outcomes.
How it connects
Connects transition probabilities to Markov decision processes, actions, rewards, policies, and Poisson distributions.

Also explained in

Compare how 7 books explain transition probability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.