Chapter 3: Grounding outputs with RAG

Introduction

This introduction explains why prompting and training alone cannot keep a chatbot current, and how RAG uses retrieved information to ground its answers.

• Prompting and live data access: Explain why basic prompting cannot answer questions that require live information, and distinguish a model’s own capabilities from external access supplied by an application.
  Why it matters: This distinction explains why a better prompt alone cannot supply today’s inventory.
• Training snapshot freshness problem: Explain why pretraining or fine-tuning on stock data cannot reliably answer changing inventory questions, and why encoding every product, location, and delivery combination is impractical.
  Why it matters: It identifies the freshness problem that motivates retrieving current information instead of storing it in model parameters.
• RAG retrieval and generation: Describe RAG’s two-stage approach: retrieve query-relevant information from up-to-date external sources, then give that context to an LLM to generate a response. Illustrate it with the TV-fit question.
  Why it matters: This is the chapter’s basic explanation of how current source information can inform a chatbot’s answer.

Key terms: basic prompting, live external data streams, application orchestrating the LLM, pretraining, fine-tune, snapshot, freshness problem, retrieval-augmented generation (RAG), real-time information retrieval, retrieved context

3.1 What is RAG?

This section explains how RAG uses retrieved information to ground LLM responses, why that helps in changing domains, and why retrieval does not guarantee reliability.

• Retrieval Grounded Responses: Explain how RAG differs from a stand-alone LLM: it retrieves passages relevant to a user’s query from external sources and uses them to inform the response. Illustrate the difference between a generic answer and one grounded in a current policy.
  Why it matters: The distinction explains why a fluent answer is not necessarily an answer based on the relevant facts.
• Freshness Without Constant Retraining: Explain why retrieving current information helps a conversational system respond when products, prices, or inventory change, without constantly retraining the language model.
  Why it matters: This is why RAG is useful for applications whose facts change faster than a model can be retrained.
• RAG Costs and Failure Modes: Describe why RAG is not a guaranteed fix for hallucinations and when putting information directly in a prompt might be simpler.
  Why it matters: Grounding helps reliability, but its costs and remaining failure modes matter when deciding whether to use it.

Key terms: RAG, retrieval component, external knowledge sources, grounding, hallucinations, real-time information retrieval, retraining, retrieval latency, embedding, indexing

3.2 RAG system architecture

Introduces the two main components of RAG architecture and their shared purpose in producing grounded responses.

• Retriever and generator roles: Describe the two key components of a typical RAG system and how they work together: retrievers obtain relevant information, and generators use it to produce responses.
  Why it matters: This basic division of work helps explain how RAG can make responses more trustworthy.

Key terms: retrievers, generators, RAG system architecture

3.2.1 The role of the retriever

This section explains how a RAG retriever finds relevant information in external sources for a user query.

• Retriever role and sources: Explain that a RAG retriever finds information relevant to each user query in external knowledge sources, and that the data is typically indexed beforehand for quick, effective retrieval.
  Why it matters: The retriever determines which external information the RAG system can bring to a user’s query.

Key terms: retriever, external knowledge sources, indexed

3.2.2 The factual generators

This section explains how a RAG generator uses a user query and retrieved information to produce a relevant, factually grounded response.

• Factual generator role: Explain that the generator is an LLM that takes the user’s query and retrieved information as input, then produces a fluent response relevant to the user’s intent and grounded in that information.
  Why it matters: It clarifies how retrieved information contributes to a RAG answer rather than serving as the answer itself.

Key terms: generators, retrieved information, user’s query, retrieved context

3.2.3 RAG system flow

This section traces how a RAG system turns a user query and retrieved knowledge into a context-informed response.

• RAG query-to-response flow: Trace the sequence from a user query through retrieval to a generated response, including what information the generator receives.
  Why it matters: This flow shows where outside information enters a RAG response.

Key terms: user query, retriever, knowledge base, generator, retrieved context

3.2.4 Example: E-commerce shopping assistant

Uses a lamp-warranty question to show how retrieval and generation work together to produce a grounded shopping-assistant response.

• Warranty query RAG flow: Trace how the shopping assistant answers the Emerald Glow Lamp warranty question: the retriever finds the lamp’s terms in available sources, and the generator uses those terms alongside the question to form its reply.
  Why it matters: This example makes the division of work between retrieval and generation concrete.
• Grounding and factual reliability: Explain why using retrieved, relevant warranty information can make the assistant’s response more trustworthy than a stand-alone language model’s response.
  Why it matters: A plausible-sounding warranty answer is more useful when it is tied to the product’s actual terms.

Key terms: retriever, generator, retrieved warranty information, grounded, hallucinations, up-to-date, verified information, stand-alone language models

3.3 Reducing hallucinations with RAG

This section introduces how grounding RAG responses in retrieved evidence helps reduce hallucinations.

• Evidence grounding against hallucinations: Explain that RAG mitigates hallucinations by grounding responses in retrieved evidence instead of relying on the model’s parametric memory alone.
  Why it matters: The distinction explains why RAG can make AI-driven conversations more trustworthy.

Key terms: RAG, hallucinations, retrieved evidence, parametric memory, grounding

3.3.1 Grounding the language model

This section explains how RAG grounds a response in retrieved information rather than relying only on a model’s training data.

• Grounding responses with retrieval: Explain that RAG retrieves relevant, up-to-date information before generating a response so the output is informed by verified sources rather than only by potentially outdated or incomplete training data.
  Why it matters: This is the basic reason retrieval can help keep responses tied to current source information.

Key terms: RAG, grounding, verified sources, training data

3.3.2 Grounding an e-commerce product query

The section contrasts a plausible but potentially wrong return-policy answer with an answer grounded in the store’s actual policy through RAG.

• Store Policy Grounding Contrast: Explain why a model answering from pretrained knowledge might get this store-credit return query wrong, and how retrieval lets the generator give a store-specific answer.
  Why it matters: The example shows why a credible-sounding answer is not enough when the correct answer depends on a store’s specific policy.

Key terms: RAG, retriever, generator, store credit, return policy

3.3.3 Advantages of RAG in reducing hallucinations

This section explains how RAG reduces hallucination risk by drawing on current, specific, traceable information from a shared knowledge base.

• Current and adaptable answers: Explain how RAG can use current information and incorporate new information without completely retraining the language model.
  Why it matters: A system can keep up when policies, products, or regulations change.
• Knowledge-base answer precision: Explain why drawing exact details from a knowledge base can make RAG answers more useful than generic responses, particularly for customers.
  Why it matters: Customers may act on an answer, so an incorrect detail can have consequences.
• Traceable authoritative sources: Explain how references to authoritative sources make a RAG answer more credible and independently verifiable.
  Why it matters: A user has more reason to trust an answer they can check against its source.
• Shared-source answer consistency: Explain why differently phrased questions should receive consistent answers at a given time, while answers can appropriately change when the documents change.
  Why it matters: Consistent answers need not mean answers remain frozen after the underlying facts change.

Key terms: current information, training data, adaptability, precise answers, knowledge base, generic responses, authoritative sources, credibility, trace, consistency

3.3.4 Critical applications

This section explains why grounding RAG responses in authoritative, current sources matters especially in high-stakes applications.

• High-stakes source grounding: Explain why RAG is valuable in health care, finance, and legal advice: small inaccuracies can have serious consequences, while grounding makes responses traceable to authoritative sources. Describe how current clinical guidelines or case law can inform answers instead of potentially outdated training data.
  Why it matters: When errors have serious consequences, readers need to understand both where an answer came from and whether its information is current.

Key terms: RAG, grounding, authoritative sources, accountability

3.3.5 Enhancing transparency

This section explains how source references make RAG responses easier to check, while emphasizing that RAG still requires oversight.

• Source references and traceability: Explain how references to the sources behind a RAG response help users and developers check it.
  Why it matters: Being able to inspect a response’s sources makes errors and missing evidence easier to spot.
• Limits of RAG reliability: Describe why source-backed RAG responses still need oversight, guardrails, and evaluation.
  Why it matters: Grounding an answer in retrieved material does not remove the need to check the answer and the material it relied on.

Key terms: references, traceability, retrieval gaps, outdated or incomplete, hallucinate, oversight, guardrails, evaluation

3.4 Data preparation and reliable indexing for RAG systems

Explains how a RAG retriever prepares and indexes data so it can find relevant information quickly.

• Four-stage RAG indexing process: Trace the indexing process from loading source data through splitting it into chunks, embedding those chunks, and storing the embeddings for retrieval. Explain why making the data searchable matters to the retriever.
  Why it matters: This sequence explains how source material becomes searchable by a RAG retriever.
• Meaning represented by embeddings: Explain what an embedding represents in this section and why converting each text chunk into numbers helps computers work with its meaning.
  Why it matters: Embeddings explain what the index stores and how related text can be found by meaning.

Key terms: indexing, DocumentLoaders, text splitters, chunks, embeddings, meaning of text

3.4.1 Introducing LangChain

The section introduces LangChain as a way to connect LLM application components and explains its limited role in the chapter’s initial RAG examples.

• LangChain building blocks for RAG: Explain what LangChain is and how connecting its components helps developers prototype a RAG system using an LLM and external knowledge.
  Why it matters: This explains why the chapter uses LangChain while assembling a RAG application.
• Limited use of LangChain: Describe how the initial examples use LangChain and distinguish that use from adopting all of its pipeline features.
  Why it matters: It keeps the focus on RAG concepts rather than treating one library as a required architecture.

Key terms: LangChain, RAG system, external knowledge, prompt templates, helper library, text splitting, document loading, RAG pipelines

3.4.2 Structuring product catalogs

The section states why embeddings are useful when preparing a product catalog for an e-commerce chatbot.

• Embeddings for product matching: Explain that embeddings help the chatbot match user queries with relevant product information from the catalog.
  Why it matters: This is the reason embeddings are useful to the catalog-based chatbot.

Key terms: embeddings, user queries, product information

3.4.3 Creating the searchable vector index

This section explains how a searchable vector index makes embedded text chunks available for efficient retrieval in a RAG system.

• Searchable vector index role: Explain that after text chunks are embedded, a RAG system stores them in a searchable vector index so it can quickly retrieve information relevant to a user query by similarity.
  Why it matters: This is how a RAG system makes embedded content searchable when answering queries.

Key terms: searchable vector index, vector store, embedded chunks, similarity

3.4.4 Implementing an index

This section explains why a FAISS index helps a chatbot retrieve relevant embedded information for its responses.

• FAISS index retrieval role: Explain that a FAISS index lets the chatbot search embedded data efficiently to retrieve relevant information for generating responses.
  Why it matters: The chatbot needs a way to find relevant information in its embedded data before generating a response.

Key terms: Facebook AI Similarity Search (FAISS), FAISS index, embedded data, high-dimensional vectors

3.5 Building an effective RAG system

This section traces how a RAG system turns a user’s question into an answer informed by retrieved data.

• Question-to-answer RAG flow: Trace how a user question leads to retrieval from an indexed knowledge base, preparation of the retrieved information as a prompt, LLM generation, and an answer informed by that data.
  Why it matters: This flow explains how retrieved information contributes to the answer rather than leaving the LLM to respond without that context.
• Embedded query matching: Explain that query processing embeds the user’s question and matches it with relevant data stored in the system.
  Why it matters: This identifies how the system connects a user’s wording to information it can retrieve.

Key terms: Question, Retrieve, Prompt, LLM, Answer, knowledge base, query processing, retrieval stage, embedding the query

3.5.1 Building an e-commerce chatbot powered by RAG

The section outlines how Clara, an e-commerce RAG chatbot, retrieves relevant information and uses it to answer a customer’s question.

• Clara’s RAG answer pipeline: Trace how Clara turns a customer query into a tailored answer: it embeds the query, finds and retrieves relevant context from its knowledge source, and passes that context with the query to an LLM.
  Why it matters: This sequence explains how the chatbot grounds its response in retrieved information.

Key terms: Clara, RAG pipeline, FAISS similarity search index, embedding model, matching contexts, LLM

3.5.2 Processing user queries

The section explains how embedding a customer query helps an e-commerce chatbot find relevant knowledge-base documents for its response.

• Query embedding for document matching: Explain that the chatbot embeds a user’s query so it can match the query with relevant information in its knowledge base, including product manuals, sizing guides, and agent chat logs.
  Why it matters: Matching the question to relevant documents gives the chatbot useful information for specific customer problems.

Key terms: query, embedding process, product data chunks, knowledge base, relevant documents

3.5.3 Crafting an accurate response using an LLM

Shows how an e-commerce chatbot combines a customer question with retrieved manual passages in a prompt so an LLM can produce a relevant answer.

• Grounded response composition: Explain how the chatbot places a customer question and relevant retrieved passages into a prompt template, then uses an LLM to turn that context into a conversational answer. Illustrate this with the bed-riser question and the answer about attachment weight and possible motor wear.
  Why it matters: It connects retrieved evidence to an answer the customer can actually use.

Key terms: prompt template, customer’s question, context, knowledge base, LLM

3.6 Evaluating and optimizing RAG systems

Introduces why RAG systems need evaluation and why assessing them is challenging.

• RAG evaluation challenge: Explain why a RAG system’s advantages do not remove the need to check its performance, and why evaluation is challenging when retrieval and generation work together.
  Why it matters: This explains why a promising RAG setup still needs its performance checked.

Key terms: RAG systems, retrieval, generation, evaluating RAG systems

3.6.1 The role of evaluator LLMs in assessing hallucinations

This section explains how evaluator LLMs assess RAG responses for hallucination risks and how their feedback supports improvement over time.

• Evaluator inputs and criteria: Explain how an evaluator LLM reviews a RAG response using the customer query, retrieved documents, and generated answer, and what it checks for.
  Why it matters: These checks distinguish an answer that sounds plausible from one supported by the retrieved material.
• Scores and improvement feedback: Describe how evaluator scores help identify responses for review and guide iterative improvements across many RAG outputs.
  Why it matters: Evaluation becomes useful when its findings lead to investigated failures and measurable changes, not just scores.
• Reliable evaluation foundations: Explain why evaluation prompts and reference examples matter to an evaluator LLM's hallucination assessments.
  Why it matters: Poor evaluation guidance or unrepresentative examples can make hallucination assessments less insightful.

Key terms: evaluator LLM, factual correctness, contextual appropriateness, hallucination indicators, nuanced scores, hallucination assessment at scale, feedback loop, evaluation prompts, reference examples, reference responses

3.6.2 Crafting evaluation datasets

This section explains how to build and maintain a telecom RAG chatbot evaluation dataset that exposes plausible but unsupported answers.

• Diverse Real-World Test Queries: Explain why the evaluation dataset should include varied user questions, such as factual, comparison, troubleshooting, billing, and hypothetical queries.
  Why it matters: A narrow test set can miss the kinds of questions that trigger hallucinations in real use.
• Vetted Answer References: Describe how to establish a factual reference for each test query using relevant, authoritative, up-to-date information from the chatbot’s knowledge base.
  Why it matters: Without trustworthy references, an evaluation cannot reliably identify factual errors.
• Plausible Unsupported Answer Examples: Explain why the dataset should include plausible but false or unsupported responses, and identify the unsupported claims in the telecom plan example.
  Why it matters: A partly true, fluent answer can still contain hallucinated details worth flagging.
• Evaluation Results and Dataset Updates: Describe how to use the dataset to flag potential hallucinations, look for recurring failure patterns, and keep the test queries and references current.
  Why it matters: Repeated evaluation is most useful when its findings drive fixes and its references remain current.

Key terms: evaluation data, query types, hallucination, authoritative answer sources, knowledge base, ground truth, vetted answer sources, hallucination examples, unsupported information, evaluator LLM

3.6.3 Practical metrics for RAG evaluation

The section introduces practical measures of RAG retrieval and answer quality, then distinguishes five RAGAS metrics.

• Practical automated RAG metrics: Distinguish retrieval precision, answer relevance, and factual accuracy by what each measures. Describe how the section scores answer relevance using question and answer embeddings.
  Why it matters: These measures reveal different ways a RAG system can succeed or fail.
• Human response quality ratings: Describe how human evaluators produce the section's overall response-quality score using a rubric and averaged ratings.
  Why it matters: Human ratings capture overall response quality rather than just one automated measure.
• RAGAS retrieved-context measures: Distinguish RAGAS faithfulness from context precision: relate the generated answer to retrieved information for one, and the retrieved information to the query for the other.
  Why it matters: The distinction separates whether the answer uses its evidence faithfully from whether useful evidence was retrieved.
• RAGAS answer quality measures: Distinguish RAGAS answer relevancy, answer correctness, and answer similarity by their respective comparisons with the query, factual answer, and reference answer.
  Why it matters: An answer can address the question, be factually right, and resemble a reference answer to different degrees.

Key terms: retrieval precision, answer-relevance score, factual accuracy, embeddings, cosine similarity, human-evaluation score, rubric, relevance, coherence, usefulness

3.6.4 Implementing RAGAS evaluation

Introduces a forthcoming RAGAS evaluation example using a sample test dataset, without showing the implementation.

3.6.5 Reducing hallucinations by filtering retrievals and using metadata

The section explains how filtering retrieved passages and keeping context concise helps a RAG system ground answers and reduce hallucinations.

• Long Context Limitations: Explain why retrieving and concatenating many long documents can make a RAG answer less reliable.
  Why it matters: More retrieved text is not necessarily better evidence for an answer.
• Semantic Relevance Filtering: Describe how semantic relevance filtering narrows retrieved passages to those that directly answer a query, using the roaming example.
  Why it matters: Filtering helps keep the model focused on evidence that answers the user’s actual question.
• Intent Matched Metadata: Explain how document metadata can restrict retrieval according to the intent of a user’s query.
  Why it matters: Metadata helps exclude documents that are on-topic but unsuitable for the kind of answer requested.
• Concise Grounding Context: Describe why relevant retrievals should also be condensed into a concise context for the LLM.
  Why it matters: Even relevant source material can be too long for the model to use effectively.

Key terms: context window, Lost in the Middle, U-shape performance curve, semantic relevance filtering, semantic similarity threshold, metadata tags, query intent, metadata-based categorization, context length optimization, extractive summarization

3.6.6 Metadata-based filtering in a telecom chatbot

Shows how a telecom RAG chatbot can extract filters from a customer’s query and use them with BM25 retrieval to find more relevant documents.

• BM25 keyword retrieval baseline: Distinguish BM25’s exact-term ranking from embedding-based semantic retrieval, and explain why BM25 is a useful baseline.
  Why it matters: This clarifies what the example’s retriever can match before metadata filters narrow its search.
• Query metadata extraction: Describe how the telecom chatbot turns a natural-language question into structured metadata filters.
  Why it matters: Turning a customer’s wording into filters lets retrieval use details such as their device and plan.
• Filtered telecom retrieval flow: Trace how extracted metadata narrows document retrieval for the roaming question and why that improves the answer’s grounding.
  Why it matters: The retrieved context is more useful when it matches the customer’s actual situation.
• Reliable filtering practices: Explain the practices the section recommends for keeping metadata-based retrieval effective.
  Why it matters: Filters help only when document tags and extracted query details are dependable without excluding too much useful material.

Key terms: Best Match (BM25), keyword-based ranking, embedding-based retrieval, QueryMetadataExtractor, PromptBuilder, OpenAIGenerator, metadata fields, structured dictionary, metadata filtering, InMemoryBM25Retriever

3.6.7 Embedding chunk sizes and model choice

This section explains how testing retrieval chunk settings and embedding models, then evaluating the whole RAG system, helps improve reliability.

• Retrieval chunk tradeoffs: Explain why chunk size and the number of retrieved chunks need balancing, and describe how testing chunk sizes, overlap, and retrieval counts can improve RAG performance.
  Why it matters: The context given to the generator must be useful without becoming noisy or incomplete.
• Task suited embedding models: Explain why a RAG system should try different embedding models rather than assume one model fits every task.
  Why it matters: A model that suits the task can contribute to better RAG performance.
• Whole system evaluation: Describe why RAG evaluation should consider both individual components and the integrated system, using datasets to test and refine configurations.
  Why it matters: A configuration that looks promising in isolation still needs evaluation as part of the complete application.

Key terms: chunk size, overlap, number of chunks retrieved, embedding model, evaluation datasets, individual components, integrated whole, evaluation and refinement

3.7 Advanced techniques for improving RAG

Introduces upcoming advanced RAG techniques as ways to address challenges and improve reliability, without explaining any technique.

3.7.1 Fine-tuning embeddings for in-domain relevance

Explains why RAG embeddings should be adapted to a domain and how to obtain and maintain in-domain training data.

• In-Domain Embedding Relevance: Explain why generic pretrained embeddings can miss domain-specific vocabulary and how in-domain fine-tuning improves RAG retrieval.
  Why it matters: Better query–document matches give a RAG system more relevant material to retrieve.
• In-Domain Dataset Maintenance: Describe the dataset examples given for fine-tuning embeddings and why retraining is needed when new document types are added.
  Why it matters: Domain adaptation must keep pace with the documents the RAG system is expected to retrieve.

Key terms: pretrained contextual embeddings, in-domain fine-tuning, retrieval accuracy, in-domain dataset, question-document pairs, synthetic questions, continuous embedding retraining

3.7.2 Incorporating metadata to reestablish context

This section explains how metadata can restore context lost when RAG retrieves documents in isolation.

• Metadata for retrieval context: Explain why isolated retrieved documents can lose context and describe two ways metadata can help: including it when creating embeddings or using it to filter results after retrieval.
  Why it matters: Metadata helps a RAG system retain clues about where retrieved content came from and what it concerns.

Key terms: metadata, embedding creation, secondary filtering stage

3.7.3 Implementing hybrid retrieval

The section explains how combining retrieval techniques can improve accuracy and mentions two other ways to combine techniques for particular needs.

• Keyword and embedding retrieval: Explain that hybrid retrieval combines keyword search with embedding similarity to improve retrieval accuracy rather than relying on either technique alone.
  Why it matters: Combining the two approaches can find relevant results that one approach alone might miss.
• Other hybrid combinations: Describe the section’s other examples: combining date and time filters with embeddings for time-sensitive retrieval, and using smaller, faster or more capable models according to query complexity.
  Why it matters: The examples show that the right combination depends on whether a query calls for timely results or greater model capability.

Key terms: hybrid retrieval, keyword search, embedding similarity, date and time filters, embeddings, cascading, query complexity

3.7.4 Enhancing queries via rephrasing and augmentation

The section distinguishes three ways to enhance a user query so retrieval can find more relevant material.

• Query enhancement approaches: Distinguish LLM query rephrasing, query augmentation, and HyDE by describing what each produces to improve retrieval.
  Why it matters: These approaches offer different ways to make a user's request more useful for retrieval.

Key terms: query rephrasing, query augmentation, ensemble retrieval effect, Hypothetical Document Embeddings (HyDE), semantic search

3.7.5 Implementing query routing

Query routing classifies a query to choose a suitable index and can assign simpler queries to smaller models, improving RAG efficiency.

• Query-specific index selection: Explain that query routing uses an initial classification to select an index optimized for the query type, such as summarization, products, or technical support.
  Why it matters: Choosing an index suited to the question helps retrieval serve different kinds of requests.
• Complexity-based model allocation: Describe how routing can send simple queries to smaller LLMs and complex queries to more capable models to improve cost efficiency.
  Why it matters: Matching model capability to the request can avoid unnecessary model cost.

Key terms: query routing, query classification, indexes, smaller LLMs, more capable models, cost efficiency

3.7.6 GraphRAG: Knowledge graph-enhanced retrieval

This section explains why cross-document questions challenge standard RAG and how GraphRAG uses a knowledge graph and summaries to support corpus-wide answers.

• Cross-document retrieval challenge: Explain why finding relevant text chunks may not be enough to answer questions that connect information across documents.
  Why it matters: This distinction explains the kind of question GraphRAG is meant to address.
• GraphRAG knowledge graph approach: Describe how GraphRAG prepares documents and what kind of retrieval this supports.
  Why it matters: The graph and summaries give retrieval a way to address connections that isolated text chunks may miss.

Key terms: Standard RAG, text chunks, relationships between entities, GraphRAG, knowledge graph, entities and relationships, community summaries, hierarchical summaries, global queries

3.7.7 When to use GraphRAG vs. traditional RAG

The section explains when GraphRAG’s cross-document connections justify its indexing overhead and sketches how a Neo4j knowledge graph is populated.

• GraphRAG tradeoff for analytical questions: Explain why GraphRAG can be preferable to traditional chunk-based RAG for complex questions that connect information across documents, despite its significant indexing overhead.
  Why it matters: This tradeoff helps you decide whether a question calls for GraphRAG rather than chunk-based retrieval.
• Knowledge graph extraction and storage: Trace the simplified example from raw text through LLM extraction to storage in Neo4j, including what Cypher MERGE does.
  Why it matters: This sketch shows what the extra graph-indexing work produces.

Key terms: GraphRAG, indexing overhead, traditional chunk-based RAG, Neo4j, entity extraction, entities, relationships, Cypher MERGE

3.7.8 Agentic RAG: Agent-controlled retrieval

This section introduces agentic RAG as agent-controlled retrieval and illustrates how it combines multiple lookups to answer a comparison question.

• Agent Control of Retrieval: Explain that agentic RAG puts an AI agent in control of retrieval, allowing it to recognize when a question requires multiple retrievals.
  Why it matters: The agent can plan retrieval around the question rather than treating every question as one lookup.
• Cross Source Comparison Answer: Trace how the example answers a question comparing electronics and clothing return policies and customer reviews: retrieve each policy separately, retrieve satisfaction data from the reviews database, and synthesize the results.
  Why it matters: The example shows why answering a compound question can require evidence from more than one retrieval.

Key terms: agentic RAG, AI agent, retrieval process, policy database, reviews database, synthesize

3.8 Building an RAG system for an e-commerce chatbot

Introduces an e-commerce RAG chatbot project and explains why LangChain helps assemble its retrieval and generation components.

• LangChain orchestration abstractions: Explain that LangChain provides higher-level abstractions for coordinating retrieval and generation, making complex RAG applications easier to build and maintain.
  Why it matters: This explains why the project uses LangChain rather than assembling every connection between retrieval and generation directly.

Key terms: LangChain, higher-level abstractions, orchestration, retrieval, generation

3.8.1 Step 1: Install the required libraries

Identifies the libraries to install for the e-commerce RAG chatbot and the role each plays.

• Chatbot foundation library roles: Describe how LangChain, PyTorch, transformers and sentence-transformers, and datasets contribute to the chatbot.
  Why it matters: Knowing each library’s role makes the chatbot’s dependencies easier to understand.
• Dense vector retrieval library: Explain why the chatbot installs faiss-cpu.
  Why it matters: This dependency supports the product-information retrieval part of the RAG chatbot.

Key terms: LangChain, torch (PyTorch), transformers, sentence-transformers, datasets, faiss-cpu, similarity search, dense vectors

3.8.2 Step 2: Import libraries

3.8.3 Step 3: Load the documents

This section explains why the product dataset matters to an e-commerce chatbot and suggests a source when no private dataset is available.

• Product dataset selection: Explain how the loaded dataset shapes the chatbot’s responses and what kind of data the section suggests using.
  Why it matters: The chatbot can draw on only the product knowledge its dataset makes available.

Key terms: dataset, product descriptions, reviews, customer Q&As, Kaggle, Amazon Reviews

3.8.4 Step 4: Document transformers

This section explains why lengthy product descriptions are split into smaller chunks before further processing.

• Product Description Chunking: Explain why lengthy product descriptions are divided into smaller text chunks: each piece should be manageable for NLP models while retaining its context.
  Why it matters: The aim is to make long descriptions easier for the model to process without stripping away context.

Key terms: document transformers, chunks, contextual information

3.8.5 Step 5: Embed text

This step explains why product descriptions are embedded and why a compact embedding model was chosen.

• Product Description Embeddings: Explain that embedding converts product descriptions into vectors that capture semantic meaning, enabling the chatbot’s retrieval component to work effectively.
  Why it matters: The chatbot needs meaningful representations of product descriptions to retrieve them effectively.
• Compact Embedding Model Choice: Explain why the section chooses sentence-transformers/all-MiniLM-L6-v2: it balances size and performance while providing good semantic understanding in a compact model.
  Why it matters: The model choice reflects a practical trade-off rather than size or performance alone.

Key terms: embedding, vector space, semantic meaning, retrieval component, sentence-transformers/all-MiniLM-L6-v2, size and performance, semantic understanding

3.8.6 Step 6: Choose a vector store

This section chooses FAISS for similarity-based product retrieval and explains its persistence limitation.

• FAISS for product retrieval: Explain why the chatbot uses FAISS as its vector store for product information.
  Why it matters: This choice determines how the chatbot retrieves relevant product information.
• FAISS index persistence: Describe what happens to a FAISS index when the process exits and what the section suggests for production.
  Why it matters: A prototype’s index should not be assumed to survive a process restart in production.

Key terms: FAISS, vector store, similarity searches, FAISS indexes, faiss.write_index(), managed vector database, persistence

3.8.7 Step 7: Prepare the LLM model

This step selects an extractive question-answering model to identify answer spans in passages retrieved for customer queries.

• Extractive model role in RAG: Explain why this step uses bert-base-uncased with AutoModelForQuestionAnswering: it pinpoints an answer span in a retrieved passage rather than generating a freeform response.
  Why it matters: Separating span selection from freeform generation clarifies what this model contributes to the chatbot.

Key terms: bert-base-uncased, AutoModelForQuestionAnswering, extractive question-answering, answer span, retriever

3.8.8 Step 8: Add a retriever

This section explains how the retriever brings query-relevant product information to the chatbot’s language model.

• Retriever role in grounding: Explain that the retriever fetches product information relevant to a query and bridges the raw data and the chatbot’s language model so responses can draw on that information.
  Why it matters: The chatbot needs relevant product data to ground its answers.

Key terms: retriever component, product information, raw data, chatbot’s language model

3.8.9 Step 9: Create a retrieval QA chain

This step combines a retriever and a QA model into a RetrievalQA chain that fetches relevant context before answering.

• RetrievalQA chain composition: Explain that a RetrievalQA chain combines a retriever with a question-answering model and fetches contextually relevant data before generating an answer tailored to the user’s query.
  Why it matters: This explains how the chatbot uses retrieved information when answering a question.

Key terms: retriever, question-answering (QA) model, RetrievalQA chain

3.8.10 Step 10: Test the chatbot

3.8.11 Step 11: Evaluate and prevent hallucinations with LangChain

This section shows how to assess whether an e-commerce chatbot’s answers are supported by retrieved documents and suggests ways to reduce unsupported answers.

• Source Token Hallucination Score: Explain how the custom hallucination_score compares a generated answer with retrieved documents, and identify the section’s conflicting descriptions of the score’s direction.
  Why it matters: Checking the score’s definition prevents a reversed interpretation of the chatbot’s results.
• Test Query Evaluation: Describe how the section proposes testing the chatbot across product questions and combining per-query hallucination scores.
  Why it matters: Testing varied questions reveals more about the chatbot than checking one answer.
• Responses To Unsupported Answers: Explain how high-hallucination test responses can guide changes to the product data, retriever, generator, or final response.
  Why it matters: The evaluation is useful when it points to a plausible source of unsupported answers and a corresponding improvement.

Key terms: hallucination_score, response tokens, retrieved source documents, grounding quality, test queries, ground-truth answers, product knowledge base, retrieval process, embedding models, similarity search

3.8.12 Challenges and adaptations of the chatbot

The section identifies practical challenges for an e-commerce RAG chatbot and emphasizes that reliable production use requires continued attention after implementation.

• Product and answer quality: Explain why an e-commerce chatbot needs current, comprehensive product data, the ability to handle varied customer questions, and an updated index to keep recommendations and answers relevant.
  Why it matters: Good recommendations depend on current information and answers that fit what customers actually ask.
• Peak demand and customer feedback: Describe the need to handle high query volumes during peak shopping times and to use customer interactions as feedback for improvement.
  Why it matters: A chatbot must work when shoppers need it most and get better from their interactions.
• RAG reliability in production: Explain that combining an LLM with dynamic retrieval can make answers more accurate and trustworthy, while production reliability still requires attention to chunking, retrieval quality, and ongoing evaluation.
  Why it matters: Adding retrieval is not, by itself, a guarantee of trustworthy answers in production.

Key terms: Data quality, diverse queries, Response accuracy and relevance, index, Scalability, Feedback mechanism, customer interactions, RAG, dynamic information retrieval, chunking strategies

3.9 Summary