liveBook · Manning

Interpretability

As explained in Retrieval Augmented Generation, The Foundational Ideas

Interpretability is the degree to which a model's information sources or decision behavior can be inspected and understood.

Interpretability is the degree to which a model’s information sources or decision behavior can be inspected and understood. Retrieval-augmented systems improve interpretability by keeping knowledge in an external, searchable corpus: users can inspect the documents retrieved for an answer and verify the result against them. This contrasts with parametric models, whose knowledge is distributed opaquely across their weights. section 3.0 section 3.4

Interpretability in retrieval-augmented systems provides:

  • Inspectable evidence: reviewers can see the exact document used to produce an answer. section 3.0
  • Operational trust: visible sources make answers easier to check than unsupported parametric predictions. section 3.4
  • Separation of knowledge and weights: an external repository can receive new information without retraining the whole model. section 3.0 section 3.4

Retrieval makes a prediction inspectable

For “The Zeppelin was invented in [MASK],” BERT without retrieval assigns Germany a 0.116 probability. Adding a retrieved sentence stating that Count Ferdinand von Zeppelin built the first rigid airship in Germany raises Germany to 0.558. The retrieved sentence both changes the prediction and supplies evidence for reviewing it. section 3.4

Listing 3.4 What retrieval buys, measured.
from transformers import pipeline
fill = pipeline("fill-mask", model="bert-base-uncased")
masked = "The Zeppelin was invented in [MASK]."
print("no retrieval:")
for r in fill(masked, top_k=3):
print(f"   {r['token_str']:<12} {r['score']:.3f}")
passage = ("Count Ferdinand von Zeppelin built the first "
"rigid airship in Germany in 1900.")  #A
print("with retrieval:")
for r in fill(passage + " " + masked, top_k=3):
print(f"   {r['token_str']:<12} {r['score']:.3f}")
 

Run the compact comparison to inspect how retrieval changes masked-token probabilities and makes the supporting passage explicit. section 3.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines inspectable evidence and contrasts it with parametric opacity, but does not develop formal interpretability methods.
Examples
Uses the Zeppelin masked-token prediction to compare unsupported probabilities with a retrieved, inspectable passage.
What you can do after
Evaluate/understand how retrieved evidence changes predictions and supports answer verification.
How it connects
Connects interpretability to retrieval-augmented generation, parametric models, retrievers, external corpora, and knowledge staleness.

Also explained in

Compare how 24 books explain interpretability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.