liveBook · Manning

Interaction information

As explained in Retrieval Augmented Generation, The Foundational Ideas

Interaction information is an information-theoretic quantity that measures how the information supplied by one variable changes when another variable is known.

Interaction information measures how much one passage’s information about an answer changes after another passage is known. For answer Y and passages X1 and X2, it is I(Y;X1) − I(Y;X1|X2). section 12.5

Read the sign as a practical retrieval diagnostic:

  • Positive values indicate redundancy: X2 makes X1 less informative because both passages communicate overlapping information. section 12.5
  • Negative values indicate synergy: X1 becomes more informative after X2 is known, so the passages answer the question together. section 12.5
  • This distinguishes graph retrieval from vocabulary-driven vector retrieval: cosine similarity can select redundant chunks, while graph traversal aims to connect complementary evidence. section 12.5 section 12.7

Evaluating a retriever ensemble

Suppose one passage describes a regulation and another identifies the affected entity, while neither answers the query alone. Their negative interaction information indicates the desired multi-hop synergy. Compare that score with marginal contribution, Multi-hop Success Rate, Hallucination Rate, and Graph Completeness before adding graph infrastructure. section 12.5 section 12.7

Table 12.4 Graph-based RAG comparison
Approach
Taxonomy
Core idea
Reasoning power
Cost (build/query)
Best use case
Microsoft GraphRAG
Global summarization
Pre-computes summaries for hierarchical communities (Leiden clusters)
Very high
High
Global summarization ("sense-making")
HippoRAG
Associative memory
Uses Personalized PageRank (PPR) to spread activation to hidden links
High
Moderate
Discovery (scientific research)
LightRAG
Hybrid (text + graph)
Uses graph structure but retrieves via efficient vector search (dual-path)
Medium
Low
Production enterprise RAG (real-time)
GRAG (Hu et al.)
KG-first
Explicit knowledge-graph traversal linearised into LLM context
High
High
Deep reasoning (compliance, legal)

Compare graph-based RAG architectures by reasoning power, construction and query cost, and intended use when choosing a retrieval design. section 12.5

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines the formula, interprets both signs, and applies the measure to retriever evaluation without deriving information-theoretic internals.
Examples
Uses complementary passages, redundant cosine-similar chunks, and retriever metrics to illustrate synergy and redundancy.
What you can do after
Evaluate retrievers by measuring whether passage pairs contribute redundancy or synergy to answers.
How it connects
Connects interaction information to information theory, vector search, graph-based RAG, retriever ensembles, and multi-hop retrieval.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.