liveBook · Manning

Markov property

As explained in Build a Robo-Advisor with Python (From Scratch)

The Markov property is the condition that the probabilities of future states depend on the current state and action but not on earlier history.

The Markov property makes the current state sufficient for predicting the next state: transition probabilities depend on the current state and chosen action, not earlier history. In goals-based investing, current wealth and asset allocation determine the possible wealth states at the next time step. section 6.2

Goals-based investing

If wealth at time t is the state and asset allocation is the action, a normal distribution can calculate the probabilities of reaching each possible wealth state at time t+1. The prior path of wealth does not enter that transition calculation. section 6.2

Listing 6.1 Creating a state space grid for goals-based investing
import numpy as np
import matplotlib.pyplot as plt
from math import log, sqrt, exp

W0 = 1000000
T = 10
nW = 101  #1
mu_stock = 0.08
mu_bond = 0.045
sig_stock = 0.20
sig_bond = 0.0

lnW = np.zeros((nW,T+1))
W = np.zeros((nW, T+1))

for t in range(T+1):
    lnW_min = log(W0) + (mu_stock-0.5*sig_stock**2)*t - 2.5*sig_stock*sqrt(t)
    lnW_max = log(W0) + (mu_stock-0.5*sig_stock**2)*t + 2.5*sig_stock*sqrt(t)
    lnW[:,t] = np.linspace(lnW_min, lnW_max, nW)  #2
W = np.exp(lnW)

plt.figure(figsize=(10, 6))   #3
plt.subplot(1,2,1)   #3
for t in range(T+1):   #3
    plt.scatter(t*np.ones(nW), W[:,t], color='r')   #3
plt.xlabel("time, $ t $")   #3
plt.ylabel("Wealth, $ W $")   #3
plt.subplot(1, 2, 2)   #3
for t in range(T+1):   #3
    plt.scatter(t*np.ones(nW), lnW[:,t], color='r')   #3
plt.xlabel("time, $ t$")   #3
plt.ylabel("Log of Wealth, $ \symbol{92}ln(W) $")  #3

A wealth-time state-space grid supplies the discrete states needed to represent Markov transitions in a financial planning model. section 6.2

How Build a Robo-Advisor with Python (From Scratch) covers it

How deep it goes
Defines the history-independence assumption and applies it to wealth transitions, while deferring transition-probability implementation to subsequent code.
Examples
Uses chessboard positions and wealth under asset allocation to illustrate history-independent transitions.
What it assumes
Assumes familiarity with states, actions, rewards, and transition probabilities within reinforcement learning.
What you can do after
Builds transition models that calculate next-period wealth probabilities from current wealth and selected asset allocation.
How it connects
Connects states, actions, transition probabilities, rewards, value functions, and reinforcement learning.

Also explained in

Compare how 3 books explain Markov property →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Robo-Advisor with Python (From Scratch). The section links point to the parts of the book it draws on.