Credit assignment is the difficulty of determining which actions in a long sequence caused an observed reward. In reinforcement learning, a single scalar often evaluates an entire response, leaving the model to infer which choices helped or hurt. section 7.3.2
Figure 7.2 The reinforcement-learning (RL) loop. The agent learns to make decisions by acting in an environment, observing a state and reward, and updating its policy to maximize cumulative reward.
The reinforcement-learning loop shows where credit assignment occurs: the agent acts, receives a state and reward, and updates its policy. section 7.3
Language-model training makes the problem especially difficult:
- A response may receive one terminal reward, so the model cannot directly distinguish useful content from incidental surface features such as formatting or sentence length. section 7.4.3 section 7.6.3
- Long or delayed feedback makes reinforcement learning noisier, more sample-hungry, and harder to stabilize than supervised fine-tuning. section 7.3.2
- Outcome supervision can reward a correct final answer even when the reasoning is wrong, because it does not identify where the reasoning failed. section 7.7.1
- Process supervision addresses the gap by scoring intermediate steps and supplying denser feedback through a process reward model. section 7.7.1
Financial summary
A financial assistant produces a 200-word summary and receives a high reward. The model cannot tell whether accuracy, faithful citations, or terse style caused the score, so it infers attribution from patterns across many scored responses. section 7.3.2
Five-step reasoning trace
An outcome reward only reports whether the final answer is good or bad. A process reward can mark steps 1 and 2 correct, step 3 incorrect, and later steps misguided, exposing the failure location. section 7.7.1

