liveBook · Manning

Transition probability

As explained in Timeless Algorithms, The Foundational Ideas

The probability of a particular next state given a current state and action.

In an inventory system, an order does not determine next-period inventory because demand varies. The transition probability P(s′ ∣ s, a) gives the probability of reaching next state s′ from current state s after action a. It represents uncertainty over possible outcomes, not a single predetermined result. section 6.5.1

In a Markov decision process, transition probabilities are considered alongside actions and rewards to evaluate possible consequences. They satisfy the Markov property: given the current state and action, the next-state probability does not depend on the complete history. When the probabilities are known, dynamic programming can use them to calculate expected future values; when they are unknown, reinforcement learning can estimate values through interaction. section 6.5.1 section 6.5.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines conditional next-state probability and its Markov property, without working through a numerical calculation.
Examples
Uses uncertain demand after an inventory order and a robot navigating an unfamiliar environment.
What you can do after
Evaluate actions by considering the likelihood and value of their possible next states.
How it connects
Connects Markov chains, Markov decision processes, dynamic programming, rewards, and reinforcement learning.

Also explained in

Compare how 7 books explain transition probability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.