Chapter 14: Advanced components for LLM applications
This introduction previews chapter 14’s extensions to RAG and agents, building on the systems introduced in chapter 13.
After this chapter you will be able to
- GraphRAG motivation and structure. Explain why semantic-similarity retrieval can miss information needed for complex questions, and describe how GraphRAG adds a knowledge graph of entities and relationships.
- Document extraction and structured output. Trace how source documents become chunks from which the local model extracts entities and relationships, and explain why the example constrains those results to a JSON schema.
- Community detection and modularity. Explain what a graph community is and trace the main decisions in the section's simplified Leiden implementation.
- Global search answer synthesis. Trace how the example's global search uses community summaries to answer a user's question.
- Global and local search. Distinguish how Microsoft’s GraphRAG generates answers with global search versus local search.
- Model support and adaptation. Explain which models Microsoft’s implementation supports directly and what adapting it for other models requires.
- Persistence and database requirements. Describe what the implementation persists and the database constraint that applies even to global search.
- CLI workspace workflow. Trace the basic CLI sequence for setting up GraphRAG, indexing a knowledge base, and running a query.
- Need for RAG metrics. Explain why BLEU and ROUGE alone are insufficient for evaluating RAG, and why evaluation must address both retrieval and generation.
- Retrieval context precision and recall. Distinguish context precision from context recall when evaluating documents retrieved for a query.
- Generated answer quality measures. Distinguish faithfulness from answer relevancy when evaluating a RAG-generated response.
- Agentic RAG retrieval decisions. Distinguish agentic RAG from traditional RAG and describe how the example can change its retrieval approach when local information is insufficient.
- Combined hybrid retrieval. Explain how the example combines BM25 and semantic retrieval into a hybrid-search tool.
- Query-directed tool selection. Trace how the final agent responds to requests for local hybrid search, web search, or both.
- Conversational memory and context. Explain how recalling relevant past interactions can improve responses, and distinguish short-term session history from long-term memory available across sessions and threads.
- Memoripy relevance and reinforcement. Describe how Memoripy finds relevant interactions and changes their importance over time.
- Memory management interaction flow. Trace the section’s Memoripy example from loading stored interactions through answering a prompt and persisting the new exchange.
What it builds on
- Combining a domain-specific SLM with RAG (chapter 13) · Assumes the reader has encountered RAG as a way to supply external context to a language model.
- Using a vector database (chapter 13) · The role of a vector database in an LLM application.
- Retrieval-augmented generation (chapter 2) · The basic distinction between retrieval and generation in RAG.
- Building an agent (chapter 13) · the basic idea of an agent
Section by section
Introduction
This introduction previews chapter 14’s extensions to RAG and agents, building on the systems introduced in chapter 13.
14.1 GraphRAG
This section explains how a local GraphRAG implementation turns source documents into a knowledge graph and community summaries, then uses those summaries to answer questions.
- GraphRAG motivation and structure: Explain why semantic-similarity retrieval can miss information needed for complex questions, and describe how GraphRAG adds a knowledge graph of entities and relationships.
Why it matters: It explains what the graph contributes beyond ordinary retrieval. - Document extraction and structured output: Trace how source documents become chunks from which the local model extracts entities and relationships, and explain why the example constrains those results to a JSON schema.
Why it matters: Reliable extraction gives graph construction usable nodes and edges. - Knowledge graph construction: Describe how the extracted and summarized elements are assembled into a graph across text chunks.
Why it matters: The composed graph is the input to community detection. - Community detection and modularity: Explain what a graph community is and trace the main decisions in the section's simplified Leiden implementation.
Why it matters: The resulting groups determine which related graph content will be summarized together. - Community summary generation: Describe how the indexed communities receive summaries and explain the main cost of producing them.
Why it matters: These indexed summaries are what the example's global search examines at query time. - Global search answer synthesis: Trace how the example's global search uses community summaries to answer a user's question.
Why it matters: This is how the graph-derived summaries contribute to an answer rather than merely being stored.
Key terms: baseline RAG, semantic similarity, knowledge graph, entities, relationships, GraphRAG, text chunks, structured outputs, JSON schema, Pydantic
14.1.1 Microsoft’s open source GraphRAG capabilities
This subsection explains Microsoft’s open source GraphRAG search options, supported models and storage, CLI workflow, and capabilities that require additional tools.
- Global and local search: Distinguish how Microsoft’s GraphRAG generates answers with global search versus local search.
Why it matters: The two search methods draw on different indexed information to answer a query. - Model support and adaptation: Explain which models Microsoft’s implementation supports directly and what adapting it for other models requires.
Why it matters: Open source code does not mean a local SLM works with the package out of the box. - Persistence and database requirements: Describe what the implementation persists and the database constraint that applies even to global search.
Why it matters: The indexing requirement affects setup even when the intended queries use only global search. - CLI workspace workflow: Trace the basic CLI sequence for setting up GraphRAG, indexing a knowledge base, and running a query.
Why it matters: The CLI provides a path from configuration to a query without writing Python code for supported options. - Missing end-to-end capabilities: State which application capabilities the open source GraphRAG solution does not provide on its own.
Why it matters: A working GraphRAG application may need tools beyond the package itself.
Key terms: global search, local search, community summaries, AI-extracted KG, embeddings, OpenAI, Azure OpenAI, open source SLMs, fork, Parquet
14.1.2 Evaluation metrics
This section explains why RAG and GraphRAG need evaluation metrics for both retrieval and answer generation, and how those metrics can be combined and implemented.
- Need for RAG metrics: Explain why BLEU and ROUGE alone are insufficient for evaluating RAG, and why evaluation must address both retrieval and generation.
Why it matters: A fluent answer or a close match to reference wording does not show that a RAG system retrieved useful evidence and used it accurately. - Retrieval context precision and recall: Distinguish context precision from context recall when evaluating documents retrieved for a query.
Why it matters: Retrieval can return relevant material yet still omit important aspects, so the two measures answer different questions. - Generated answer quality measures: Distinguish faithfulness from answer relevancy when evaluating a RAG-generated response.
Why it matters: An answer can be grounded in the documents yet fail to address the question, or address the question without being faithful to its evidence. - Domain-weighted RAG score: Explain how the listed metrics can form an overall RAG score and why their contributions should be weighted.
Why it matters: A single score should reflect which kinds of retrieval and answer quality matter most for the application. - Ragas support limitation: State the section’s implementation caveat for using Ragas with local SLMs or custom models.
Why it matters: The availability of metric definitions does not mean the named library directly supports the book’s local-model setting.
Key terms: BLEU, ROUGE, precision, recall, RAG metrics, context precision, context recall, retrieved documents, faithfulness, answer relevancy
14.2 RAG + Agentic AI
The section builds an agentic RAG example that lets an agent choose among local keyword search, semantic search, hybrid search, and web search for a user’s query.
- Agentic RAG retrieval decisions: Distinguish agentic RAG from traditional RAG and describe how the example can change its retrieval approach when local information is insufficient.
Why it matters: The agent’s ability to adapt retrieval is what makes this more than a fixed RAG pipeline. - Local BM25 keyword search: Explain what the example’s BM25 tool searches and how the agent uses it for a keyword query.
Why it matters: Keyword search gives the agent a way to retrieve relevant passages from the local documents. - Embedding-based semantic search: Describe how the example prepares its LanceDB knowledge base and retrieves chunks with the semantic-search tool.
Why it matters: Semantic search gives the agent a local retrieval option based on embeddings rather than keyword ranking alone. - Combined hybrid retrieval: Explain how the example combines BM25 and semantic retrieval into a hybrid-search tool.
Why it matters: Combining keyword and semantic results is the local retrieval capability used in the final multi-tool agent. - Query-directed tool selection: Trace how the final agent responds to requests for local hybrid search, web search, or both.
Why it matters: This demonstrates the value of agentic RAG when useful information is spread across sources.
Key terms: agentic RAG, traditional RAG, retrieval, AI agents, BM25, BM25Retriever, BM25SearchTool, document chunks, LanceDB, embeddings
14.3 Long- and short-term memory management
This section explains how short- and long-term memory can add relevant past interactions to LLM prompts and shows how Memoripy manages that memory.
- Conversational memory and context: Explain how recalling relevant past interactions can improve responses, and distinguish short-term session history from long-term memory available across sessions and threads.
Why it matters: The distinction tells you which previous context an application can bring into a new conversation. - Memoripy relevance and reinforcement: Describe how Memoripy finds relevant interactions and changes their importance over time.
Why it matters: Relevance and reinforcement explain why memory retrieval is more selective than simply replaying a transcript. - Memory management interaction flow: Trace the section’s Memoripy example from loading stored interactions through answering a prompt and persisting the new exchange.
Why it matters: The flow shows both how memory informs an answer and how a new answer becomes available for later interactions. - Memory storage and fields: Describe the storage choices and the roles of the principal fields in a stored memory entry.
Why it matters: These fields connect the stored exchange to the retrieval and reinforcement behavior described in the section.
Key terms: short-term memory, long-term memory, context, Memoripy, embeddings, concepts, cosine similarity, spreading activation, decay, MemoryManager
14.4 Summary
Concepts explained in this chapter
- knowledge graphA structured graph representation of entities and relationships used here to enrich RAG with explicit connections among concepts.
- community summaryA high-level summary of a group of related graph entities used as an intermediate unit for search and response generation.
- retrieval-augmented generationAn application system that combines retrieval from external sources with text generation to ground outputs in outside knowledge.
- Leiden algorithmA community-detection algorithm that improves graph clustering by optimizing modularity through iterative node movement and aggregation.
- BM25A lexical information-retrieval ranking algorithm that scores documents using term frequency, inverse document frequency, and document length.
- agentic RAGA retrieval-augmented generation setup that uses an AI agent to plan, select, and combine retrieval actions across one or more sources before generation.
- hybrid searchA retrieval method that combines keyword-based and vector-based search to improve retrieval quality.
- AI agentAn autonomous software entity that can invoke models or tools as part of a larger system.
- spreading activationA retrieval method that propagates activation from labeled source nodes through linked nodes in an associative network.
- working memoryConversation memory that keeps recent interactions within a session so they can be reused immediately.
- GraphRAGA retrieval-augmented generation approach that uses knowledge graphs and community summaries to support multihop reasoning over document collections.
- Louvain algorithmA community-detection algorithm that iteratively moves nodes and aggregates communities to improve a clustering objective.
- communityA group of graph nodes that are more densely connected to each other than to the rest of the graph.
- hierarchical clusteringA clustering approach that builds communities through repeated refinement and aggregation of grouped items.
- CodeAgentA smolagents agent pattern that generates Python code to invoke tools and solve a task step by step.
- smolagentsA Hugging Face library for building AI agents with code-based or tool-calling workflows.
← Chapter 13: Creating end-to-end LLM applications
Chapter 15: Test-time compute and small language models →
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.