liveBook · Manning

Interpretability

As explained in AI Governance

The ability to provide an understandable account of how an AI system reaches its decisions or outputs.

An AI-assisted decision can leave a customer without a meaningful explanation if the system has no record of the model version, retrieved documents, or transformations applied to the output. Interpretability addresses the need for an understandable account of an outcome; in practice, that requires distinguishing an explanation from the evidence available to check it. section 6.4 section 6.4.1

Transparency reconstructs what happened: the inputs, configuration, information, tools, outputs, and people involved. Explainability asks why a particular outcome happened to a particular person. A complete trace may not explain the outcome, while a coherent rationale may lack records that allow it to be checked. section 6.4 section 6.4.1

Build an operational basis for reviewing decisions:

  • Assemble a decision record from the execution pipeline, retrieval layer, and review interface. Include the model version, prompt configuration, retrieved sources, input, output, and human review as appropriate to the use case. section 6.4
  • Expose supporting evidence, such as retrieved documents, policy clauses, and extracted facts, alongside known limitations. Reviewers can then assess the rationale rather than accept a black-box judgment. section 6.4.1
  • Correlate prompts, retrieval, tool calls, approvals, and outputs with observability tooling. Instrumentation helps reconstruct events, but auditor-ready, tamper-proof evidence bundles typically require additional work. section 6.4 section 6.4.1
Table 6.3 The Decision Record (conceptual schema). Each row is one automated or AI-assisted decision. Fields are grouped by transparency layer. In practical scenarios, operational and regulatory information may live in separate tables or collections.
Layer
Field Name
Description
Provider
model
Identifier or alias for the base model version. Contains {provider, model_name, model_tag, api_release_date, model_card_url}.
System
system_prompt_version
Hash or ID for the current prompt or system configuration
System
retrieval_sources
IDs or hashes of documents shown to the model. Contains {doc_id, snippet_offset, source_version}
System
input
The raw user input that triggered the decision. Contains {user_prompt_redacted_id, user_prompt_hash}
System
output
The raw model output before post-processing. Contains {outcome (one of: “approve”,“deny”,“request_info”,“escalate”), structured_key_facts:[], policy_refs:[]}
System
postprocessing_id
List of transformations applied after model response
Explainability
rationale_explanation
Human-readable justification, constructed via template or retrieval
Explainability
evidence
List of factors that influenced the decision, derived from business rules, retrieval sources, or structured extraction. Contains {factor_id, factor_value, source, actionable_guidance}
Explainability
confidence_flag
Optional confidence or evidence score
Reviewer
review_id
Identifier for the human review. Contains {reviewer_id, outcome (one of: “approve”,“deny”,“request_info”,“override”), review_timestamp, override_explanation}
Customer
access_log:[]
Logs access requests. Contains {user_id (pseudonymized), timestamp, request_id}
Regulator
bundle_id
ID tying this record to quarterly export bundles
Regulator
lawful_basis
GDPR basis for retention and processing
Regulator
record_hash
Cryptographic hash of the full record, logged for audit integrity

The decision record schema groups fields by transparency layer, providing a starting point for a correlated case file. section 6.4

A claim rationale is not proof of reasoning

A simulated trace recommends a partial denial because repair costs are below the deductible and damage falls under a policy exclusion. The rationale gives a reviewer concrete claims to check against the policy and evidence, but it does not establish that those factors actually caused the model’s output. Model-generated traces can be incomplete, strategic, or reconstructed after the fact. section 6.4.1

This distinction matters especially for autonomous agents: visible “thinking” does not guarantee a genuine explanation, even when the agent’s actions affect individuals. Keep human review connected to evidence and use override patterns to prioritize prompt, retrieval, and documentation fixes. section 2.6.4 section 6.4.1

How AI Governance covers it

How deep it goes
Distinguishes explanation from event reconstruction and details review controls, while stopping short of reliable access to black-box model reasoning.
Examples
Uses a claim-denial rationale, a decision-record schema, and human-override metrics to illustrate review and improvement.
What you can do after
Evaluate decision rationales against supporting evidence and recognize when a trace cannot establish the model’s actual reasoning.
How it connects
Connects transparency, observability, human review, retrieval, and chain-of-thought prompting.

Also explained in

Compare how 24 books explain interpretability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.