liveBook · Manning

Interpretability

As explained in Machines that Think

Interpretability is the degree to which a model's internal reasoning or outputs can be understood by people.

Interpretability is the extent to which people can understand a model’s internal workings or outputs. It matters when a decision must be examined, challenged, or justified—not merely observed. section 1.4.4 section 7.4.1

Learned models can discover patterns that are difficult to specify in advance, but their decisions may be distributed across many numerical relationships. Inputs and outputs remain visible even when the influences between them are hard to trace. This is known as the black box problem. Opacity can complicate safety checks, accountability, bias detection, and debugging, especially in high-stakes settings. section 1.4.4

Explainable AI (XAI) uses several approaches to make model behavior easier to inspect:

  • Intrinsically interpretable models, such as decision trees and rule-based systems, expose more of their logic directly, though they may be less flexible on complex tasks. section 7.4.1
  • Feature attribution estimates which inputs contributed most to an output. It can describe importance across decisions or focus on one result. section 7.4.1
  • Counterfactual explanations identify changes that would have produced a different outcome. section 7.4.1
  • Surrogate models approximate a complex system with a more readable one. section 7.4.1

Inspecting a loan decision

An explanation can help reveal whether a loan model relied on relevant financial factors or on proxies for socioeconomic background. A counterfactual might identify changes that would have led to approval. section 7.4.1

Figure 7.5 Explainable AI techniques. When a model produces an impactful decision, explainability methods try to make the path from input to output easier to inspect. Feature attribution estimates which parts of the input contributed most to the result, counterfactual explanations identify small changes that would have altered the outcome, intrinsically interpretable models use simpler structures that can be inspected directly, and surrogate models approximate a complex system with a more readable one.

Compare four ways to make the path from input to output easier to inspect. section 7.4.1

Interpretability is one dimension of evaluation rather than a substitute for others. In language classification, for example, precision and interpretability both matter; strength on one measure can conceal weaknesses elsewhere. section 2.6.6

How Machines that Think covers it

How deep it goes
Defines the black box problem and surveys explanation methods, without implementation detail.
Examples
Uses loan decisions and hiring rankings to illustrate relevant factors and potentially questionable signals.
What you can do after
Evaluate and understand model explanations when assessing decisions, potential bias, and debugging needs.
How it connects
Connects interpretability to explainable AI, bias, feature attribution, counterfactual explanations, and surrogate models.

Also explained in

Compare how 24 books explain interpretability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.