Return is the reward an agent accumulates over a sequence of decisions. Rather than judging an action only by its immediate payoff, reinforcement learning aims to maximize cumulative reward, often in expectation when outcomes are uncertain. A policy is a rule for choosing actions; the goal is to find one with high expected long-term reward. section 12.1 section 12.2
What counts as a good sequence depends on the decision problem:
- In a Markov decision process, actions affect future states, so their later consequences matter. section 12.1
- In a multi-armed bandit, the agent repeatedly chooses among arms without changing future states. It seeks a high total payoff while balancing exploration against exploitation. section 12.1.6
Ninety slot-machine pulls
Suppose three arms have mean payoffs of 8, 10, and 5, and the agent makes 90 choices. Always choosing the mean-10 arm would yield an expected total payoff of 900; sampled payoffs will vary. A simulation instead compares strategies using average regret: on each trial, it subtracts the sampled payoff from the best arm’s mean payoff, then averages those differences. Regret is related to return, but is not the return itself. The exploit-only example selects using the supplied means; it does not demonstrate learning those means from scratch. section 12.1.6
def ucb(c):
num_plays = np.zeros(K) #1
sum_rewards = np.zeros(K) #2
total_regret = 0 #3
for i in range(K): #4
reward = sample_payoff(i)
num_plays[i] += 1
sum_rewards[i] += reward
for t in range(K, num_trials): #5
ucb_values = sum_rewards / num_plays + c * np.sqrt(np.log(t) / num_plays) #6
selected_machine = np.argmax(ucb_values)
reward = sample_payoff(selected_machine)
num_plays[selected_machine] += 1
sum_rewards[selected_machine] += reward
optimal_reward = max_reward
regret = optimal_reward - reward
total_regret += regret
average_regret = total_regret / num_trials
return average_regret
The UCB listing tracks reward sums and play counts separately for each arm to guide future selections. Those per-arm sums are not the agent’s total return across all choices. section 12.1.6
