A reward signal is numerical feedback received after an agent takes an action. It tells the agent how that action turned out, but the objective is usually to maximize reward accumulated over time, not simply the next reward. The agent learns through repeated interaction: choose an action, observe feedback, and adjust its decision-making policy. section 12.1 section 12.1.2
In a Markov decision process, the agent acts in a state and receives both a reward and a new state. An action can therefore affect later choices and rewards. A multi-armed bandit strips away those transitions: each choice produces a payoff, but the agent remains in one state. A contextual bandit adds information that can make a different action preferable in a different context. section 12.1 section 12.1.2 section 12.1.6
One point at a time: CartPole
A controller moves a cart left or right to keep its pole upright. It earns +1 for each time step the pole remains upright, so lasting longer increases the episode’s total reward. In the Python A2C implementation, each environment step returns a reward; training adds it to the episode total and uses it in an advantage calculation to update the actor and critic. Plotting episode rewards against episode number makes progress visible. section 12.3
The feedback must suit the decision problem:
- For three slot machines, each arm has a payoff distribution rather than a fixed payoff. Sampling rewards allows exploration-only, greedy, ε-greedy, and upper-confidence-bound strategies to be compared by average regret. The toy greedy strategy is given the arms’ means, so its result is not a demonstration of learning them from scratch. section 12.1.6
- For truck selection, a reward function scores a combination of size, engine, and tires, then adds random noise to represent uncertain conditions. A contextual-bandit loop selects a combination and feeds its observed reward back into training. section 12.5
Check that the implemented reward actually reflects the desired choice. The shown truck reward function accepts a route context but calculates its score only from truck-option values and noise; as written, its feedback cannot express different truck preferences for city and interstate routes. Noisy rewards also make one observed score an unreliable verdict on an action. section 12.5
def reward_function(shared_context, size_index, engine_index, tire_index):
size_value = [0.8, 1.0, 0.9] #1
engine_value = [0.7, 0.9, 1.0] #2
tire_value = [0.9, 0.8, 1.0, 0.95] #3
reward = (
size_value[size_index]
* engine_value[engine_index]
* tire_value[tire_index]
) #4
noise_scale = 0.05 #5
noise_value = np.random.normal(loc=0, scale=noise_scale) #5
reward += noise_value #5
return reward ` #6
Inspect how option values are combined and noise is added—and verify whether the supplied context affects the returned reward. section 12.5
