Interpretability is the ability to understand and explain a model’s behavior through its input features, predictions, and internals such as structure and parameters. It matters when a prediction alone is insufficient: a diabetes diagnosis can inform treatment decisions, while explanations can build trust, reveal failure cases, and expose data bias during debugging. section 9.0 section 9.1
A useful first distinction is between glass-box and black-box models. Glass-box models have intuitively readable internals, such as decision trees expressed as decision rules or generalized linear models with interpretable feature weights. Complex models may remain black boxes even when their parameters are available, because those parameters do not yield a human-usable explanation. section 9.1.1 section 9.6
Figure 9.1 With black-box machine-learning models, we can only use the input-output pairs to analyze and explain model behavior. The model internals in a black box are either unavailable or aren’t directly interpretable. With glass-box machine-learning models, in addition to input-output pairs, the model internals are also intuitively interpretable.
Compare what can be inspected in each model type: black-box analysis relies on input-output behavior, while glass-box analysis can also use readable internals. section 9.1.1
Interpretability methods differ by the scope of the explanation they seek:
- Global methods explain the model’s overall decision-making behavior and broadly relevant factors. For black-box models, permutation feature importance measures the performance change after shuffling a feature, while partial dependence plots summarize a feature’s effect by marginalizing other variables. section 9.6
- Local methods explain one prediction for one example. LIME is model-agnostic: it generates weighted examples near the target example, then fits an interpretable linear surrogate that approximates black-box behavior in that neighborhood. section 9.4.1
Surrogate models offer another route to explaining a black box. Train an interpretable model, such as a decision tree or GLM, on the black-box model’s predictions rather than the original labels. A global surrogate is useful only to the extent that it mimics the original model, so its fidelity must be balanced against the simplicity needed for human interpretation. For a tree, limiting leaf nodes can improve readability but may reduce fidelity. section 9.3.3 section 9.6
Figure 9.11 The procedure to train a global surrogate model from the predictions of a black-box model. Both models are trained on the same surrogate training examples. However, the surrogate model is trained on the predictions of the black-box model, so that it can learn to mimic its predictions. If the black box and surrogate make the same prediction, then the surrogate can be used to explain the black-box model’s prediction.
Follow the global-surrogate workflow: use shared examples, label them with black-box predictions, and fit a glass-box model to mimic those predictions. section 9.3.3
Explaining an ensemble prediction
An XGBoost model trained on Bank Marketing data can be approximated globally by a decision-tree surrogate. The surrogate is trained on XGBoost predictions, then constrained to 14 leaf nodes for inspection. Its usefulness depends on whether it remains faithful enough to the ensemble while staying small enough to understand. For an individual prediction instead, LIME can show which features push a subscription prediction toward or away from a positive outcome. section 9.3.3 section 9.4.1
