liveBook · Manning

Interpretability

As explained in AI Model Evaluation

Explainability is the degree to which the basis for a model's output or judgment can be understood and inspected by people.

Interpretability is the ability to trace, interpret, and understand model outputs rather than treating them as a black box. It helps people reason about why a model behaves as it does. section 3.0

A model change can improve one metric while harming another. Comparing multiple metrics makes those trade-offs visible, helping teams decide whether a change is acceptable rather than relying on a single score. section 2.3.2

Make an evaluation easier to interpret by:

  • Defining the decision, comparison data, and desired output, such as a label, score, or ranking. section 9.4
  • Making judgments repeatable with clear task framing, prompts, reasoning expectations, and output formats. section 9.5
  • Inspecting a judge’s reasoning when a score looks wrong; it can help reveal a misunderstanding or a defensible disagreement, but does not prove the judgment is correct. section 10.3.1

Rating summaries

Instead of asking an LLM to judge summary “quality,” specify criteria such as clarity, factuality, and redundancy, provide the source text, and choose an interpretable output such as a score or ranking. section 9.4

Figure 2.5 illustrates a dashboard that displays multiple metrics simultaneously, enablings engineers to see how changes to a model affect different performance aspects at once. Key features include side-by-side comparisons between baseline and new model performance and visual indicators showing whether changes are positive or negative, based on acceptable thresholds to avoid interpretation errors.

Compare a baseline and a new model across metrics, with indicators showing whether changes meet acceptable thresholds. section 2.3.2

How AI Model Evaluation covers it

How deep it goes
Shows how dashboards and reasoning traces aid inspection, without treating explanations as proof that judgments are correct.
Examples
Uses metric dashboards, criteria-based summary ratings, and judge reasoning traces as inspection aids.
What you can do after
Evaluate model changes by comparing metrics and inspecting judgments or reasoning when results look wrong.
How it connects
Links interpretability to diagnostic evaluation, offline evaluation, and LLM-as-a-judge evaluation.

Also explained in

Compare how 24 books explain interpretability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.