In an inventory system, an order does not determine next-period inventory because demand varies. The transition probability P(s′ ∣ s, a) gives the probability of reaching next state s′ from current state s after action a. It represents uncertainty over possible outcomes, not a single predetermined result. section 6.5.1
In a Markov decision process, transition probabilities are considered alongside actions and rewards to evaluate possible consequences. They satisfy the Markov property: given the current state and action, the next-state probability does not depend on the complete history. When the probabilities are known, dynamic programming can use them to calculate expected future values; when they are unknown, reinforcement learning can estimate values through interaction. section 6.5.1 section 6.5.3






