Interpretability

As explained in Knowledge Graphs and LLMs in Action

The provision of understandable evidence or reasoning that allows people to inspect and validate an artificial intelligence system's outputs.

Interpretability makes AI outputs inspectable by exposing the evidence, relationships, and reasoning behind them. Knowledge graphs support this by organizing extracted facts into verifiable structures rather than leaving answers inside an opaque model. section 6.4 section 1.3

A practical interpretability workflow can:

• Trace an answer through ontology relationships and information paths. section 7.4.2
• Expose supporting documents and structured facts for inspection. section 6.4
• Represent conflicting sources and validate or challenge apparent connections. section 6.4 section 7.4.2

Diabetes and pancreatic islets

A graph can show why type 1 diabetes and islets of Langerhans are connected, distinguish synonymous or ambiguous entity names, and let users follow the relevant ontology path into supporting documents. section 7.4.2

Figure 1.5 Summary of how LLMs and KGs can complement each other. Inspired by [11].

Use the complementarity diagram to inspect how language models add natural-language querying while knowledge graphs preserve traceable, structured reasoning. section 1.3

How Knowledge Graphs and LLMs in Action covers it

How deep it goes

Explains traceable graph paths, conflicting sources, and ontology-based discovery without detailing model-internal computation.

Examples

Uses diabetes, pancreatic islets, ontology paths, document retrieval, and graph-based views of unusual disease co-occurrences.

What you can do after

Evaluate and validate AI findings by following structured relationships, inspecting sources, and comparing competing interpretations.

How it connects

Connects interpretability with knowledge graphs, ontologies, large language models, GraphRAG, and retrieval-augmented generation.

liveBookmanning.com

Read Knowledge Graphs and LLMs in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Outlier Detection in PythonExplainable artificial intelligence (XAI) comprises methods and systems that make an artificial-intelligence model's predictions or decisions understandable to people.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentInterpretability is the degree to which humans can understand how a model produces its outputs or decisions.
• 

  liveBookmanning.com
  Financial AI in PracticeThe degree to which people can understand the basis of a model’s predictions or decisions.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningModel interpretability, or explainability, is the degree to which people can understand and explain a model's behavior and predictions.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionInterpretability is the degree to which people can understand how an AI system produces its outputs.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringModel interpretability is the degree to which a model's behavior and predictions can be understood by people or analysis methods.
• 

  liveBookmanning.com
  Machine Learning System DesignThe property of a model or automated decision-making system whose behavior and rationale are visible or understandable to relevant people.
• 

  liveBookmanning.com
  First-Party Data ActivationThe degree to which a model’s outputs and the factors producing them can be understood by people.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIThe degree to which a model's operation and predictions can be understood by people.
• 

  liveBookmanning.com
  Inside AIInterpretability is the degree to which a model’s decisions or outputs can be understood and explained by people.
• 

  liveBookmanning.com
  Transformers in ActionExplainability is the degree to which a model's outputs and the factors behind them can be understood or audited.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionInterpretability is the degree to which a machine-learning model's learned representations, predictions, and decisions can be understood by humans.
• 

  liveBookmanning.com
  Causal AIExplainable AI is the development of artificial intelligence systems whose predictions or decisions can be interpreted or explained to people.
• 

  liveBookmanning.com
  How Large Language Models WorkAn approach to artificial intelligence that produces information intended to make a model's outputs or decision process understandable to people.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe degree to which a model's behavior can be understood and interpreted by humans.

Compare how 16 books explain interpretability →

Related concepts

• LIMEuses · LIME is a model-agnostic explanation technique that approximates a model near an individual input with an interpretable surrogate to explain its prediction.
• SHAPuses · SHAP is a feature-attribution method that explains model outputs by assigning input features contributions based on Shapley values.
• counterfactual explanationuses · A counterfactual explanation is a minimally changed hypothetical input whose model outcome differs from that of the original input.
• surrogate modeluses · A surrogate model is an approximate model that estimates the behavior or outputs of a more complex, costly, or otherwise unavailable model, process, or function.
• accuracycontrasts with · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.
• transparencypart of · Transparency is the degree to which underlying content is visible through an element or relevant information, processes, and decisions are accessible and understandable to observers.

This explainer was generated by Manning's knowledge graph from the text of Knowledge Graphs and LLMs in Action. The section links point to the parts of the book it draws on.