liveBook · Manning

Credit assignment

As explained in LLM Evaluation and Alignment, The Foundational Ideas

The credit assignment problem is the difficulty of determining which actions or decisions caused an observed reward, especially across long sequences.

Credit assignment is the difficulty of determining which actions in a long sequence caused an observed reward. In reinforcement learning, a single scalar often evaluates an entire response, leaving the model to infer which choices helped or hurt. section 7.3.2

Figure 7.2 The reinforcement-learning (RL) loop. The agent learns to make decisions by acting in an environment, observing a state and reward, and updating its policy to maximize cumulative reward.

The reinforcement-learning loop shows where credit assignment occurs: the agent acts, receives a state and reward, and updates its policy. section 7.3

Language-model training makes the problem especially difficult:

  • A response may receive one terminal reward, so the model cannot directly distinguish useful content from incidental surface features such as formatting or sentence length. section 7.4.3 section 7.6.3
  • Long or delayed feedback makes reinforcement learning noisier, more sample-hungry, and harder to stabilize than supervised fine-tuning. section 7.3.2
  • Outcome supervision can reward a correct final answer even when the reasoning is wrong, because it does not identify where the reasoning failed. section 7.7.1
  • Process supervision addresses the gap by scoring intermediate steps and supplying denser feedback through a process reward model. section 7.7.1

Financial summary

A financial assistant produces a 200-word summary and receives a high reward. The model cannot tell whether accuracy, faithful citations, or terse style caused the score, so it infers attribution from patterns across many scored responses. section 7.3.2

Five-step reasoning trace

An outcome reward only reports whether the final answer is good or bad. A process reward can mark steps 1 and 2 correct, step 3 incorrect, and later steps misguided, exposing the failure location. section 7.7.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains response-level and step-level attribution, but defers the update mechanics and reward-source design to related treatments.
Examples
Uses a financial summary, formatting bias, and a five-step math trace to show ambiguous and localized credit.
What it assumes
Assumes basic reinforcement-learning rewards and policies while contrasting them with token-level supervised fine-tuning signals.
What you can do after
Diagnose reward ambiguity and evaluate whether process-level supervision is needed for long or multi-step responses.
How it connects
Connects reinforcement learning, supervised fine-tuning, reward models, outcome supervision, process supervision, and reward hacking.

Also explained in

Compare how 2 books explain credit assignment →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.