A transition probability gives the chance of moving from a current state at time t to a particular state at time t+1 after an action. In reinforcement learning, these probabilities describe the uncertain dynamics that connect states, actions, and rewards. section 6.2
The calculation assumes the Markov property: the current state contains the information needed for the next-state distribution, so earlier history is not used. For goals-based investing, the current state is wealth and the action is an asset allocation; the resulting distribution gives probabilities for next-period wealth states. section 6.2
Discrete wealth-state probabilities
Start with a grid of possible wealth values. For a selected current wealth, time, and allocation, compute portfolio mean and volatility, then evaluate a normal-distribution CDF at the midpoints between adjacent next-period wealth states. The difference between adjacent CDF values is the probability of landing in that state’s interval. The lowest interval begins near zero and the highest extends to infinity. A zero-volatility allocation needs special handling because the usual standardization divides by volatility. This normal-return assumption can be replaced with another distribution, including one with fat tails. section 6.2.1
Listing 6.2 Computing transition probabilities
from scipy.stats import norm
import warnings
warnings.filterwarnings('ignore')
def compute_mu_sigma(a):
mu = a/(nA-1)*mu_stock + (1-a/(nA-1))*mu_bond
sig = a/(nA-1)*sig_stock
return mu, sig
def compute_midpoints(W):
W_midpts = np.zeros((nW+1,T+1))
W_midpts[0,:] = 0.000001
W_midpts[nW,:] = np.inf
W_midpts[1:nW,:] = (W[:nW-1,:]+W[1:nW,:])/2
return W_midpts
def compute_transition_probs(w, t, a, W_midpts):
mu, sig = compute_mu_sigma(a)
W_tplus1_dist = np.zeros(nW)
cdfs = norm.cdf(np.float64((np.log(W_midpts[:,t+1]/w)
- (mu-0.5*sig**2))/sig)) #1
W_tplus1_dist = cdfs[1:nW+1] - cdfs[0:nW] #2
return W_tplus1_dist
This implementation computes allocation-specific portfolio parameters, constructs wealth-bin midpoints, and returns adjacent CDF differences as a probability vector over next wealth states. section 6.2.1
Dynamic programming uses the probability vector to calculate an action’s expected next value by taking its dot product with next-period state values. Backward induction then selects the allocation with the greatest expected value at each wealth-time state. section 6.2.1
When spending is also an action, consumption is subtracted from current wealth before computing the next-state distribution. Expected value then combines current consumption utility with discounted future value, allowing allocations and consumption levels to be compared together. section 6.3.2
