A reward signal is a scalar evaluation of how desirable an action, state, outcome, or completed trajectory is. Reinforcement learning uses it as the optimization target: an agent observes a state, chooses an action, receives reward, and updates its policy toward behavior that earns more reward over time. The signal need not describe every detail of success; it operationalizes relative goodness in a form an optimizer can use. section 3.1.1 section 3.1 section 10.3
In a finite-horizon RL problem, reward is commonly written r(s_t, a_t). The policy objective is to maximize expected discounted return: the sum of rewards along trajectories, with discount factor gamma balancing immediate against later rewards. A reward signal therefore supplies local evaluations, while return measures the total consequence of a policy's sequential decisions. section 3.1
Equation 3.2
\[\label{eq:rl_opt} {\max_\pi \; \mathbb{E}_{\tau \sim p_{\pi}} \left[ \sum_{t=0}^{T-1} \gamma^t r(s_t, a_t) \right].}\]This objective makes the operational role of reward explicit: optimize a policy for expected discounted cumulative reward over trajectories. section 3.1
Reward design depends on what produces the signal:
- Environment-defined reward can directly encode a task condition. A thermostat receives +1 when temperature is within 2 degrees of 70°F and 0 otherwise. section 3.1.1
- Per-step reward can favor sustained task success. In CartPole, each step spent balanced and on-track receives reward 1; failure ends the episode. section 3.1.2
- Learned reward can turn preference data into an optimization signal. In RLHF, a reward model scores text quality from human preference relations, then ranks generated completions for RL updates. section 1.3 section 5.0
A reward model is especially useful when the desired behavior is difficult to specify with fixed rules. It compresses complex human judgments into one score per input sequence, playing the learning-signal role that an environment reward plays in standard RL. This convenience also makes the reward model a proxy objective: optimization follows what the model scores, not necessarily every aspect of the underlying human preference it was meant to represent. section 5.0 section 5.9
Pairwise reward scores for language-model completions
Given prompt x, a Bradley–Terry reward model assigns scalar scores rθ(yc|x) and rθ(yr|x) to a chosen completion yc and rejected completion yr. Training increases their difference so the chosen answer is more likely to be preferred: P(yc > yr|x) = sigmoid(rθ(yc|x) − rθ(yr|x)). Only score differences matter, so adding the same constant to all scores changes no pairwise preference probability. In implementation, a causal language model can pool the final non-padding-token hidden state and pass it through a linear head to emit one scalar reward. section 5.1 section 5.2
Reward signals also support estimates of future reward. A Q-function maps each state–action pair to a scalar estimate of future reward, while a value function maps states to the same kind of estimate. Temporal-difference learning uses newly collected data to update such estimates and address credit assignment, connecting immediate signals to longer-run outcomes. section 10.4 section 10.4
The scalar form is powerful but reductive. Preference reward models use a useful approximation of human preferences rather than a universal account of them, and reward-model implementations vary with training libraries and data pipelines. Alternative reward-model forms can score final outcomes or individual reasoning steps, depending on the task. section 5.1 section 5.2 section 5.9
