A reward signal is numerical feedback assigned after an action. Positive rewards reinforce similar choices; negative rewards discourage them. In reinforcement learning, an agent adjusts its policy to maximize cumulative reward over time. For financial planning, the agent is an investor acting in stock and bond markets, with wealth and time forming the state and asset allocation forming the action. section 6.2
Reward design determines what “success” means. A goals-based plan can pay only at the horizon: reward 1 when terminal wealth exceeds the goal and 0 otherwise. The resulting value function is expected cumulative reward and, under this binary payoff, directly equals the probability of reaching the goal. Starting with $1 million, a value of 0.68 means a 68% goal-achievement probability under the optimal policy. section 6.2.1
Rewards can instead be received each period. In a spending problem, period reward is utility of consumption, while wealth evolves after consumption and investment returns. Future rewards are weighted by the subjective discount factor γ: γ=1 gives equal weight to present and future rewards, while γ=0 values only the immediate reward. Terminal wealth is consumed and assigned its utility at the final period. section 6.3.2 section 6.2.2
In dynamic programming and Q-learning, rewards enter decision values differently:
- Dynamic programming works backward from terminal rewards, evaluating each asset allocation by transition probabilities and future-state values. section 6.2.1
- Q-learning updates a state-action estimate using current reward plus discounted maximum next-state Q, minus the current estimate. section 6.2.2
- Spending Q-learning adds utility(consumption) as the immediate reward while selecting both allocation and consumption. section 6.3.2
Binary goal reward versus consumption reward
For a $2 million terminal goal, an investor receives 1 only when final wealth exceeds $2 million; otherwise the reward is 0. For ongoing spending, each chosen consumption amount receives utility immediately, and the algorithm trades that utility against discounted future utility and remaining wealth. section 6.2.1 section 6.3.2
Listing 6.9 Updating \(Q\) for the spending problem
def Update_Q(w_idx, t, Q):
if np.random.uniform() < epsilon:
a = np.random.randint(nA)
c = np.random.randint(nC)
else:
AC_array = Q[w_idx,t,:,:]
ac = np.argwhere(AC_array == AC_array.max()) #1
if (len(ac)>1):
a,c = ac[np.random.choice(len(ac))]
else:
a,c = ac[0]
cons = (c+1)/(1.5*nC)*W[w_idx,t]
W_tplus1_idx = transition(W[w_idx,t]-cons, t, a) #2
Q[w_idx,t,a,c] += alpha*(utility(cons) +
gamma*Q[W_tplus1_idx,t+1,:,:].max()-Q[W_idx,t,a,c]) #3
return W_tplus1_idx, t+1, Q
Inspect the Q update that combines utility of current consumption with discounted future Q values while choosing an allocation-consumption pair. section 6.3.2
