liveBook · Manning

Causal path

As explained in Machines that Think

A causal pathway is the sequence of underlying factors and processes through which inputs produce an outcome.

A causal path is the sequence of factors and processes through which inputs shape an outcome. For language models, a generated explanation may be plausible without faithfully describing that path. section 7.2.1 section 7.4.2

Several methods can reveal partial evidence about what influenced an output, but each has limits:

  • Token attribution estimates which prompt tokens or contextual inputs influenced an answer; token-level influence does not necessarily provide a clear human-readable reason. section 7.4.2
  • Attention analysis inspects relationships between tokens, but attention to a token does not establish that it caused the output. section 7.4.2
  • Probing tests whether information is encoded in internal activations, while mechanistic interpretability traces computations associated with behavior; causal tracing remains difficult at large scale. section 7.4.2
  • Chain-of-thought faithfulness analysis checks whether written reasoning reflects the factors that actually shaped an answer. section 7.2.1 section 7.4.2
Figure 7.6 Explainability methods for language models. In this example, the model produces a hiring recommendation and justifies it by referring to the candidate’s test score, but the surrounding analyses raise a different possibility: the output may have been influenced by the “School A” token. Token attribution estimates which prompt tokens most shaped the answer, attention analysis inspects relationships between tokens, representation analysis tests whether bias is encoded in internal activations, mechanistic interpretability traces internal pathways associated with the behavior, and chain-of-thought faithfulness checks whether the written reasoning matches what actually drove the response.

A hiring recommendation cites a test score, while other analyses raise the possibility that a “School A” prompt token influenced the answer. section 7.4.2

How Machines that Think covers it

How deep it goes
Compares several analysis methods and their limits, noting that tracing causal computations remains difficult at large scale.
Examples
Illustrates hidden prompt cues shifting answers and a hiring recommendation potentially influenced by a “School A” token despite its stated test-score rationale.
What you can do after
Evaluate generated explanations against evidence from input attribution, representation analysis, or internal-computation tracing.
How it connects
Connects causal pathways to token attribution, attention analysis, representation analysis, mechanistic interpretability, and chain-of-thought faithfulness analysis.

Also explained in

Compare how 2 books explain causal path →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.