Chapter 13: Knowledge graph–powered retrieval-augmented generation

Introduction

The introduction contrasts the ease of building NLP applications with LLMs against the work still needed to make them useful in enterprise production.

• Shift in NLP development: Explain how GPT-3.5 and prompt engineering changed the effort required to build NLP applications.
  Why it matters: This shift explains why LLMs changed how practitioners approach NLP applications.
• Limits of production LLMs: Explain why powerful LLMs alone are not presented as a complete solution for enterprise production use, and name the developments this realization led to.
  Why it matters: It frames the gap between a working LLM demonstration and an enterprise-ready application.

Key terms: GPT-3.5, prompt engineering, downstream task, AI agents, LLM operations (LLMOps), production enterprise-grade scenarios

13.1 AI agents

This section defines AI agents, explains why complex tasks call for capabilities beyond simple LLM question answering, and illustrates how specialized agents can work together.

• AI agent autonomy: Explain what makes an AI agent different from a traditional program: it performs tasks by interacting with its environment and can adapt its decisions to changing conditions.
  Why it matters: This distinction explains why agents are considered for tasks that cannot be handled as a fixed sequence of steps.
• Demands beyond simple questions: Describe why answering a straightforward factual question is not the motivating example for an expensive LLM-based system, and identify the harder demands the section lists: multistep reasoning, deep relational understanding, and access to current or private data.
  Why it matters: These demands motivate systems that do more than ask an LLM for an answer from its existing training.
• Specialized multiagent roles: Describe how the content-writing example divides a task among a Researcher, Writers, and a Reviewer, and explain how their specialized roles contribute to a shared goal.
  Why it matters: The writing example makes the section’s idea of collaborating agents concrete.

Key terms: AI agents, autonomy, adaptability, multistep reasoning, deep relational patterns, knowledge cutoff, multiagent system, Researcher, Writers, Reviewer

13.2 Chatting with the LLM

The section builds a simple conversational LLM agent and shows why a follow-up can improve an answer while an out-of-the-box model still needs relevant data for specific responses.

• Conversational agent memory: Describe how the example supports a conversation: it uses a pretrained LLM and keeps the full history of questions and answers so the user can ask a follow-up.
  Why it matters: Keeping the exchange lets a user clarify an earlier question rather than start over.
• Specific answers and supplied data: Explain what the initial question and clarification demonstrate about answer specificity, and why an out-of-the-box model cannot be relied on for specific answers grounded in expected data it has not seen.
  Why it matters: A better-worded follow-up can help, but it does not replace giving the model the data needed for a grounded answer.

Key terms: pretrained LLM, memory, messages, message history, follow-up question, out-of-the-box models, specific answers

13.3 Challenges in the production environment

This section explains why LLM-based agents face production challenges and why access to current external information can help with one of them.

• Plausible but fabricated answers: Explain why an LLM can produce a convincing but inaccurate answer, especially when asked about something absent from its training data.
  Why it matters: A convincing answer is not necessarily an accurate one.
• Knowledge freshness and retrieval: Explain the knowledge-cutoff problem and why an agent might use tools to retrieve current external information.
  Why it matters: External information can help an agent answer questions about developments newer than the model’s training.
• Answer transparency concerns: Describe what is missing when an LLM gives a coherent answer without showing how it was produced.
  Why it matters: Enterprise users may need more than a polished answer to judge whether to rely on it.
• Sensitive data and access: Describe the privacy and access-control concerns that arise when using LLMs with organizational data.
  Why it matters: A useful agent must not expose sensitive information to people who should not see it.
• Deployment costs and bias: Describe the resource and ethical concerns the section raises about deploying LLMs.
  Why it matters: Production decisions must account for both the resources models consume and the harm their outputs may cause.

Key terms: hallucinations, next token, freshness, knowledge cutoff, external sources, knowledge graph, transparency, information sources, reliability, confidence level

13.4 Chatting with the AI about private data

This section motivates an AI interface that lets people ask questions about a private knowledge graph without learning graph-query tools.

• Specialized knowledge gap: Explain why an LLM needs access to domain-specific private data while retaining its language understanding and general knowledge.
  Why it matters: It explains why the section turns to private data rather than relying on the pretrained LLM alone.
• Archive influence network: Describe what the Rockefeller Archive Center KG captures and how its unpublished information supports questions about research-funding influence.
  Why it matters: The archive shows the kind of question that requires access to private, connected information.
• Accessible AI interface: Explain the proposed value of an AI interface to the KG and identify the components the section names for that approach.
  Why it matters: It frames the AI interface as a way to make the private KG useful beyond people skilled with graph tools.

Key terms: LLM, specialized domains, private data, Rockefeller Archive Center, KG, influence network, proprietary data, AI interface, AI agents, pretrained LLM

13.4.1 Retrieval-augmented generation

This section explains how RAG grounds LLM answers in retrieved external information, illustrates vector-based retrieval, and notes why human oversight remains necessary.

• External context for RAG answers: Explain how RAG combines an LLM with question-relevant information retrieved from an external source to generate an answer, and why this can reduce hallucinations and provide up-to-date information.
  Why it matters: Retrieved context lets an LLM answer questions that its pretrained knowledge alone may not answer reliably.
• Vector-based document retrieval: Trace how the section’s text-based RAG example prepares documents as embeddings and retrieves text chunks similar to a user’s question.
  Why it matters: This retrieval path explains where the text supplied as RAG context comes from.
• Limits of RAG grounding: Explain why RAG cannot guarantee correct answers and why the section recommends human feedback, validation, or supervision.
  Why it matters: Retrieved evidence improves reliability, but an answer may still need human checking.

Key terms: retrieval-augmented generation, RAG agent, external data source, grounding, tools, text chunks, embeddings, vector database, same model, most similar text chunks

13.4.2 Vector-based RAG limitations

This section explains why vector-based RAG can retrieve misleading or incomplete context and introduces knowledge graphs as a way to address those limitations.

• Fragmented document context: Explain how retrieving separate document chunks can hide relationships across documents or split information needed for an answer.
  Why it matters: An answer can fail even when the needed information exists in the documents.
• Embedding representation limits: Describe why dense, precomputed document embeddings may miss fine-grained or domain-specific meaning and adapt poorly to changing terminology.
  Why it matters: Semantic similarity is only as reliable as the representations used to calculate it.
• Large corpus search tradeoff: Explain the accuracy tradeoff that vector search may face as a document corpus grows.
  Why it matters: The choice of search method can affect whether useful documents are found.
• Noisy and missing retrieval: Distinguish irrelevant retrieved context from missing relevant context, and use the Lauritsen example to explain why semantic similarity does not ensure an entity match.
  Why it matters: A capable LLM cannot reliably repair irrelevant or incomplete retrieved evidence.
• Knowledge graph context benefits: Explain the proposed role of a knowledge graph in RAG and why its integrated, editable knowledge can improve confidence in the system.
  Why it matters: The section's proposed alternative makes the source of AI context easier to integrate, maintain, and inspect.

Key terms: context fragmentation, document chunks, multihop relationships, chunking strategy, embeddings, dense vector, domain-specific nuances, static (precomputed) vectors, scalability, approximate search algorithms

13.4.3 Graph RAG

The section explains how a knowledge graph can supply and select context for RAG, then illustrates a graph-guided document retriever in a simple agent.

• Graph structure and document context: Distinguish a text-attributed graph from a text-paired graph, and explain how document metadata in the KG can help select retrieval context.
  Why it matters: The graph’s links and metadata give retrieval more to work with than document text alone.
• Knowledge graph context retrieval: Explain how a KG retriever can provide context to an LLM, including the two retrieval approaches described here.
  Why it matters: This is how graph facts can become grounding context for an answer.
• Graph-guided document selection: Explain how a text-paired KG narrows document retrieval and trace the section’s relation-focused retriever tool from inputs to returned documents.
  Why it matters: Graph-guided selection can keep irrelevant documents out of the LLM’s context.
• Combined retrieval across sources: Describe why an agent may combine KG and document retrieval, and explain the order of retrieval in the section’s transaction example.
  Why it matters: One source may contain the identity needed to search another source effectively.

Key terms: text-attributed graph, text-paired graph, metadata, KG retriever, typed relationships, schema, Cypher, subgraph, KG-enhanced document retriever, precanned Cypher query

13.4.4 Reasoning agents

This section explains how a ReAct agent coordinates Graph RAG tools and what makes the illustrated agent more reliable or limited.

• ReAct tool selection loop: Trace how the ReAct agent uses an original question to choose a tool, act, assess the result, and either try another tool or answer.
  Why it matters: The loop explains how one agent can coordinate tools without a predetermined sequence.
• Agent configuration and testing: Explain how tool descriptions, prompts, and repeated testing affect the agent’s behavior.
  Why it matters: Tool selection and consistent behavior depend on how the agent is configured and tested.
• Illustrative system limitations: Distinguish the illustrated KG-powered Graph RAG agent from a production-ready system and name the improvements the section suggests.
  Why it matters: It prevents mistaking a demonstration for a complete deployment design.

Key terms: ReAct, Reason and Act, tools, contextual information, structured ReAct agent, tool names, tool descriptions, default prompts, KG-powered Graph RAG, document re-ranking

13.4.5 Let’s chat with our KG

This section tests a Graph RAG agent on archive questions, showing how it selects retrieval tools and why a knowledge graph helps answer questions spanning multiple documents.

• Question-dependent retrieval choices: Trace how the agent chooses retrieval tools for questions about what researchers said: it first finds people who talked about Dorothy M. Wrinch in the KG and then retrieves relevant documents, but can use the KG-enhanced document retriever directly when two people are specified.
  Why it matters: The example shows how the agent finds documentary evidence for an answer rather than relying on the LLM alone.
• Cross-document aggregate answers: Explain why structured KG content can answer an aggregate question spanning documents, and contrast this with relying on vector search for a complete set of relevant documents.
  Why it matters: It explains why searching for similar documents alone may not supply a complete answer to a question about shared facts.
• RAG reliability improvements: Describe the two suggested ways to improve RAG reliability: have an LLM check and potentially correct a generated Cypher query, or rerank initially selected documents to improve relevance and limit context size.
  Why it matters: These suggestions identify two different places where a RAG answer can be improved before it is produced.

Key terms: Graph RAG agent, KG retriever, KG-enhanced document retriever, Thought/Action/Observation, structured KG content, aggregate question, vector search, Cypher, distraction phenomenon, self-correction loops

13.5 Summary