Feature attribution assigns importance scores to input features, indicating how they contribute to a model prediction. For language models, perturbation-based attribution removes or switches input features and observes changes in the target prediction. Captum exposes this workflow through FeatureAblation and LLMAttribution, producing token-level scores that can be visualized as input-to-output heat maps. section 10.2.1
Apply feature attribution through several complementary workflows:
- Wrap a language model with LLMAttribution, pass tokenized input and a target string to attribute(), and inspect the resulting feature scores. section 10.2.1
- Visualize token attribution with a heat map to compare how input tokens influence generated target tokens. section 10.2.1
- Replace token-level input with TextTemplateInput when word-piece segmentation makes the explanation difficult to read. section 10.2.1
- Use baselines to represent absent feature states and compare predictions against a chosen reference input. section 10.2.1
- Use LIME for model-agnostic local explanations in text classification, sentiment analysis, or named entity recognition. section 10.2.2
Explaining generated interests
Given a prompt describing Nicole as a data scientist in Zurich, attribute the target “hiking, skiing, and playing the piano.” The resulting heat map can show that “Data scientist” contributes positively to “playing” but negatively to “hiking.” A template can keep “data scientist” together instead of exposing fragmented word pieces. section 10.2.1
Explaining sentiment
Run a sentiment classifier on “This movie was absolutely amazing!”, provide its probability function to LimeTextExplainer, and display word-level weights for the local positive prediction. section 10.2.2
Listing 10.6 Using FeatureAblation and LLMAttribution from Captum
fa = FeatureAblation(model) llm_attr = LLMAttribution(fa, tokenizer)
Inspect the minimal Captum setup that wraps FeatureAblation with LLMAttribution before requesting language-model explanations. section 10.2.1
Listing 10.12 Using LIME to understand local model decisions
def predict_proba(texts): #1 inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True) outputs = model(**inputs) probabilities = torch.softmax(outputs.logits, dim=-1).detach().numpy() return probabilities explainer = LimeTextExplainer(class_names=['negative', 'positive']) #2 input_text = "This movie was absolutely amazing!" #3 explanation = explainer.explain_instance(input_text, predict_proba, num_features=10) explanation.show_in_notebook() #4
Inspect the LIME workflow that supplies prediction probabilities, explains one text instance, and renders the explanation. section 10.2.2
