A causal path is the sequence of factors and processes through which inputs shape an outcome. For language models, a generated explanation may be plausible without faithfully describing that path. section 7.2.1 section 7.4.2
Several methods can reveal partial evidence about what influenced an output, but each has limits:
- Token attribution estimates which prompt tokens or contextual inputs influenced an answer; token-level influence does not necessarily provide a clear human-readable reason. section 7.4.2
- Attention analysis inspects relationships between tokens, but attention to a token does not establish that it caused the output. section 7.4.2
- Probing tests whether information is encoded in internal activations, while mechanistic interpretability traces computations associated with behavior; causal tracing remains difficult at large scale. section 7.4.2
- Chain-of-thought faithfulness analysis checks whether written reasoning reflects the factors that actually shaped an answer. section 7.2.1 section 7.4.2
Figure 7.6 Explainability methods for language models. In this example, the model produces a hiring recommendation and justifies it by referring to the candidate’s test score, but the surrounding analyses raise a different possibility: the output may have been influenced by the “School A” token. Token attribution estimates which prompt tokens most shaped the answer, attention analysis inspects relationships between tokens, representation analysis tests whether bias is encoded in internal activations, mechanistic interpretability traces internal pathways associated with the behavior, and chain-of-thought faithfulness checks whether the written reasoning matches what actually drove the response.

A hiring recommendation cites a test score, while other analyses raise the possibility that a “School A” prompt token influenced the answer. section 7.4.2

