A multi-armed bandit is a sequential decision problem in which an action produces an outcome without a multi-step state-transition sequence. In RLHF, one prompt and its generated completion form a complete episode: the completion is the action, and the process normally does not produce a next prompt. section 3.1.3
The reward is assigned to the entire completion rather than individual generated tokens, so the setup is response-level and usually uses no discounting (γ = 1). This makes RLHF implementation materially different from traditional multistep reinforcement learning. section 3.1.3
Prompt-to-completion decision
Sample a prompt x, generate a completion y from the policy, and score the resulting prompt–completion pair with a learned reward or preference model. The complete interaction is one rollout rather than a chain of state–action transitions. section 3.1.3
Table 3.1 Key differences between standard RL and RLHF for language models
| Aspect | Standard RL | RLHF (language models) |
| Policy | Learned from scratch (random initialization) | Fine-tuned from a pretrained language model |
| Reward signal | Environment reward function \(r(s_t,a_t)\) | Learned reward/preference model \(r_\theta(x,y)\) (prompt \(x\), completion \(y\)) |
| State transition | Yes: dynamics \(p(s_{t+1}\mid s_t,a_t)\) | Typically no: prompts \(x\) sampled from a dataset; the completion does not define the next prompt |
| Action | Single environment action \(a_t\) | A completion \(y\) (a sequence of tokens) sampled from \(\pi_\theta(\cdot\mid x)\) |
| Reward granularity | Often per-step / fine-grained | Usually response-level (bandit-style) over the full completion; usually no discounting (\(\gamma = 1\)) |
| Horizon | Multistep episode (\(T>1\)) | Often single-step (\(T=1\)) (although agentic training can be multiturn) |
Compare standard reinforcement learning with RLHF across policy initialization, reward source, state transitions, action structure, reward granularity, and horizon. section 3.1.3
