Captum is an open-source PyTorch library for interpreting model predictions. It exposes attribution methods that help inspect how language models prioritize input information and produce text, supporting analysis of bias, errors, and transparency. section 10.2 section 10.2.1
Captum separates attribution approaches by the access they require:
- Perturbation-based methods repeatedly evaluate a black-box model with varied inputs and do not require access to model weights. section 10.2.1
- Gradient-based methods estimate attribution by backpropagating gradients and therefore require access to model weights. section 10.2.1
- Attribution results can be visualized across layers, attention heads, or individual tokens to inspect which information influenced generated output. section 10.2
Attribute generated text with FeatureAblation
Load Llama 3, create a prompt about Nicole, and generate a continuation. Wrap FeatureAblation with LLMAttribution, represent the prompt with TextTokenInput, and request attribution for the target phrase “hiking, skiing, and playing the piano.” Plot the result as a token-level heat map. TextTemplateInput can instead separate prompt values, while baselines such as alternate names, locations, occupations, and pronouns support refined comparisons.
