Interpretability is the extent to which people can understand a model’s internal workings or outputs. It matters when a decision must be examined, challenged, or justified—not merely observed. section 1.4.4 section 7.4.1
Learned models can discover patterns that are difficult to specify in advance, but their decisions may be distributed across many numerical relationships. Inputs and outputs remain visible even when the influences between them are hard to trace. This is known as the black box problem. Opacity can complicate safety checks, accountability, bias detection, and debugging, especially in high-stakes settings. section 1.4.4
Explainable AI (XAI) uses several approaches to make model behavior easier to inspect:
- Intrinsically interpretable models, such as decision trees and rule-based systems, expose more of their logic directly, though they may be less flexible on complex tasks. section 7.4.1
- Feature attribution estimates which inputs contributed most to an output. It can describe importance across decisions or focus on one result. section 7.4.1
- Counterfactual explanations identify changes that would have produced a different outcome. section 7.4.1
- Surrogate models approximate a complex system with a more readable one. section 7.4.1
Inspecting a loan decision
An explanation can help reveal whether a loan model relied on relevant financial factors or on proxies for socioeconomic background. A counterfactual might identify changes that would have led to approval. section 7.4.1
Figure 7.5 Explainable AI techniques. When a model produces an impactful decision, explainability methods try to make the path from input to output easier to inspect. Feature attribution estimates which parts of the input contributed most to the result, counterfactual explanations identify small changes that would have altered the outcome, intrinsically interpretable models use simpler structures that can be inspected directly, and surrogate models approximate a complex system with a more readable one.

Compare four ways to make the path from input to output easier to inspect. section 7.4.1
Interpretability is one dimension of evaluation rather than a substitute for others. In language classification, for example, precision and interpretability both matter; strength on one measure can conceal weaknesses elsewhere. section 2.6.6























