Interpretability is the ability to inspect why a language model produces an output, rather than judging that output alone. It supports debugging and the investigation of possible bias or errors. It also depends on transparency beyond the model’s calculations: knowing what went into pretraining can help trace the origins of downstream problems. section 10.2 section 10.1
Two complementary ways to make model behavior easier to examine are:
- Attribute an output to inputs. Visualization can show how information is prioritized at the layer, attention-head, or token level. Captum offers perturbation-based methods, which evaluate changed inputs without needing model weights, and gradient-based methods, which use backpropagation and require access to the weights. section 10.2 section 10.2.1
- Audit information about the model. The foundation model transparency index uses 100 indicators covering development resources, model characteristics, and applications; its scores can inform model selection. This assesses disclosure, not the cause of an individual output. section 10.3.1
- Trace retrieved evidence. In retrieval-augmented generation (RAG), known, retrievable sources make it easier to backtrack and audit what information was available for an answer. section 5.4.2
Attribute a generated continuation with Captum
Load a Llama 3 8B Instruct model, provide a prompt about a person’s location, occupation, and interests, and generate a continuation. To investigate the continuation “hiking, skiing, and playing the piano,” wrap Captum’s FeatureAblation in LLMAttribution. Supply the prompt as TextTokenInput and the continuation as the target, then plot token attributions. Feature ablation removes input features and observes how that changes the target prediction. A TextTemplateInput can instead group prompt fields; supplying baselines makes the comparison explicit. Read the resulting heat maps as attribution for this target and input setup, not as a complete account of the model’s decisions. section 10.2.1
Listing 10.7 Getting the attribution result of your LLM
inp = TextTokenInput( eval_prompt, tokenizer, skip_tokens=[1]) target = "hiking, skiing, and playing the piano." attr_res = llm_attr.attribute(inp, target=target)
The attribution call pairs a tokenized prompt with the specific continuation whose input contributions are being examined. section 10.2.1
Listing 10.10 Plotting distilled attribution results of your LLM
inp = TextTemplateInput(
template="{} lives in {}, {}
and is a {}. {} personal interests include",
values=["Nicole", "Zurich", "Switzerland", "data scientist", "Her"],
baselines=["Elon", "Los Angeles", "United States",
"businessman and investor", "His"],)
attr_res = llm_attr.attribute(inp, target=target)
attr_res.plot_token_attr(show=True)
The template and baselines show how to compare grouped prompt fields against specified alternatives. section 10.2.1
