Reinforcement learning trains an agent to choose actions through trial and error. The agent interacts with an environment, receives rewards or penalties, and adjusts a policy: rules for selecting actions in particular situations to maximize cumulative reward over time. It suits decisions whose current choices alter later options. section 6.2 section 6.0
In goals-based investing, the investor is the agent and stock and bond markets form the environment. A state can combine wealth W and time t; an action can be an asset allocation. Wealth is discretized into a finite, time-indexed grid, with a wider range at later dates to reflect growing dispersion of possible outcomes. section 6.2
A practical reinforcement-learning setup specifies:
- State: the variables relevant to a decision, here wealth and discrete time. section 6.2
- Action: a choice that changes the state, such as the percentage of wealth invested in stocks versus a risk-free bond. section 6.2
- Reward: feedback that encourages or discourages future actions and drives policy improvement. section 6.2
- Episode: one forward run from t=0 through terminal time T, repeated to learn from many trajectories. section 6.2.2
Q-learning learns a value for each state-action pair, Q(S_t,A_t), rather than only a state value. Starting from initialized Q values, it moves forward through episodes and updates the chosen pair using the immediate reward plus the discounted maximum estimated Q value at the next state. The learning rate controls the size of each update; the discount rate γ expresses how future rewards are weighted. section 6.2.2 section 6.2.2
Learning an investment policy
Initialize a Q array across wealth states, times, and allocations. At each step, sometimes select a random allocation for exploration; otherwise select an allocation with the highest current Q value. Simulate the next wealth state, update Q, and repeat across many episodes. The resulting policy can choose allocations conditional on both wealth path and time rather than prescribing one fixed glide path. section 6.2.2 section 6.6
For multiperiod planning, reinforcement learning offers an alternative to dynamic programming. Dynamic programming works backward from a terminal state and requires a transition probability distribution; reinforcement learning can learn without that distribution. It can also be useful when many states and actions make dynamic programming difficult, though results remain dependent on the modeled rewards, transitions, and planning assumptions. section 6.6 section 6.0
The same framework can jointly optimize spending and asset allocation using utility functions, and can extend planning models with mortality tables for longevity risk. Potential decisions include Social Security claiming, pension payout choice, and annuity purchase; additional extensions include income, health expenses, taxes, housing, bequests, and alternate utility functions. section 6.6
Listing 6.6 Generating episodes
from tqdm import tqdm
n_episodes = 1000000
epsilon = 0.5
alpha = 0.1
gamma = 1.0
Q = np.zeros((nW, T+1, nA))
for j in range(nW):
if W[j, T]>G:
Q[j, T, :] = 1
for i in tqdm(range(n_episodes), desc="Running..."):
W_idx = 0
t = 0
while t<=(T-1):
W_idx, t, Q = Update_Q(W_idx, t, Q)
The episode loop makes the learning process concrete: initialize terminal Q values, repeatedly start from the initial wealth state, and update Q forward in time. section 6.2.2
