Chapter 6: RAG fundamentals with ChromaDB
The chapter introduces meaning-based search across documents and identifies RAG and vector stores as foundations for the Q&A systems it will build.
After this chapter you will be able to
- Meaning-Based Document Search. Explain that semantic search finds document fragments that match the meaning of a question rather than merely matching its keywords, and that this is also called Q&A over documents or querying a knowledge base.
- Meaning-Based Information Retrieval. Explain that semantic search interprets a query’s meaning to retrieve relevant document fragments from a document store, and may then generate a natural-language answer.
- Single-document Q&A components. Describe how a document, prompt, and LLM-based chatbot work together to answer a question about one document. Explain that the prompt supplies both the question and the document as context, from which the LLM selects information and synthesizes an answer.
- Chatbot follow-up advantage. Distinguish the chatbot from a Q&A engine using the incomplete Paestum answer: a person can ask a follow-up without resending the original text, whereas an engine called by a client system returns one answer. Explain that the chatbot retains the exchange by passing conversation history as additional context.
- Unsupported answers and safer prompts. Explain why a chatbot's answer to a question not covered by the supplied text may be unreliable, even when it sounds confident. Describe the section's safer prompt: require answers based only on the supplied text and ask the chatbot to say when the answer is unknown.
- Focused Knowledge-Base Context. Explain why an enterprise Q&A chatbot should retrieve only content relevant to a question instead of putting all company content in the prompt. Describe how this lets a user ask naturally without supplying the background documents each time.
- RAG stages and roles. Explain what retrieval, augmentation, and generation mean in this Q&A design, and distinguish content ingestion from question answering.
- Chunk and embedding preparation. Trace how source documents become searchable chunks and explain why the text is split and embedded.
- Question-time retrieval and generation. Trace a user question through embedding, similarity search, prompt construction, and LLM answer generation.
- Vector stores and embeddings. Explain what a vector store holds and why embeddings are useful for indexing content.
- Similarity search by vector distance. Describe how a vector store finds content similar to a query and what that similarity means.
- Vector distance and similarity search. Explain that vector search compares vectors using distance calculations, and that KNN and the more scalable ANN search use such calculations to find similar vectors.
- Dense and sparse search vectors. Distinguish dense vectors, whose dimensions are mostly nonzero and were used for image-embedding search, from sparse vectors, whose values are mostly zero and were used for exact-word lexical search.
- Updates and ingestion queries. Distinguish early vector libraries from vector databases in their ability to update data and run similarity searches while importing new data.
- Text and embedding linkage. Explain how storing original content separately from embeddings complicated early vector libraries, and how vector databases simplified that relationship.
- Limits of vector store comparisons. Explain why a comparison of popular vector stores is only a starting point for exploring the market.
- Chroma collection and ingestion. Describe how the example creates an in-memory Chroma client and a collection, then stores manually split Paestum text chunks with IDs and source metadata. Explain that Chroma generates embeddings from the supplied text.
- Semantic search and distance. Explain how querying the populated collection retrieves the closest text chunks and how embedding distances indicate which chunks are more relevant to the question.
- Retrieval versus answer generation. Distinguish Chroma’s retrieval of relevant text chunks from an LLM’s generation of a formulated answer using the question and retrieved chunks.
- Retrieval-backed chatbot answer flow. Trace how the proposed chatbot answers the Paestum question: the user asks only the question, the chatbot retrieves content from ChromaDB, and GPT-5-nano receives that content along with the original question.
- Reusable content retrieval. Describe how the section reuses semantic-search code in a function and checks its result against a previously asked question.
- Question and context prompt. Describe how the user's question and context retrieved from the vector database are combined in a prompt and submitted to the LLM.
- Safer Q&A prompt instructions. Explain how the recommended RAG prompt aims to mitigate hallucinations, including what it tells the LLM to do when the retrieved context does not support an answer.
- Chatbot answer boundaries. Describe how the basic chatbot answers questions using text imported into the vector database and the section’s claim about information absent from that database.
Section by section
Introduction
The chapter introduces meaning-based search across documents and identifies RAG and vector stores as foundations for the Q&A systems it will build.
- Meaning-Based Document Search: Explain that semantic search finds document fragments that match the meaning of a question rather than merely matching its keywords, and that this is also called Q&A over documents or querying a knowledge base.
Why it matters: This distinction explains what the document Q&A system is meant to retrieve. - RAG and Vector Store Roles: State that RAG is presented as the design pattern behind semantic search systems and that a vector store is a key component of those systems, without attributing implementation details to this introduction.
Why it matters: These roles orient the reader before the chapter explains the components in detail.
Key terms: semantic search, Q&A over documents, querying a knowledge base, RAG, vector store, semantic search systems
6.1 Semantic search
Defines semantic search and explains how meaning-based retrieval differs from keyword matching.
- Meaning-Based Information Retrieval: Explain that semantic search interprets a query’s meaning to retrieve relevant document fragments from a document store, and may then generate a natural-language answer.
Why it matters: This distinction helps you separate finding relevant content from turning it into an answer. - Meaning Versus Keyword Matching: Distinguish semantic search from traditional keyword-based search when a query and a relevant result use different words.
Why it matters: It explains why meaning-based search can find relevant material that exact-word matching misses.
Key terms: semantic search, query, document fragments, document store, keyword-based searches
6.1.1 A basic Q&A chatbot over a single document
This section uses a single-document Q&A example to explain how a chatbot answers from supplied context, supports follow-up questions, and can produce unreliable answers when the text lacks the requested information.
- Single-document Q&A components: Describe how a document, prompt, and LLM-based chatbot work together to answer a question about one document. Explain that the prompt supplies both the question and the document as context, from which the LLM selects information and synthesizes an answer.
Why it matters: These are the basic pieces behind the section’s document-based Q&A example. - Chatbot follow-up advantage: Distinguish the chatbot from a Q&A engine using the incomplete Paestum answer: a person can ask a follow-up without resending the original text, whereas an engine called by a client system returns one answer. Explain that the chatbot retains the exchange by passing conversation history as additional context.
Why it matters: Follow-up questions can repair an incomplete response without making the user supply the document again. - Unsupported answers and safer prompts: Explain why a chatbot's answer to a question not covered by the supplied text may be unreliable, even when it sounds confident. Describe the section's safer prompt: require answers based only on the supplied text and ask the chatbot to say when the answer is unknown.
Why it matters: A plausible answer is not necessarily grounded in the document or safe to trust.
Key terms: document, prompt, context, LLM-based chatbot, synthesize, Q&A engine, chatbot, stateful session, history of the conversation, additional context
6.1.2 A more complex Q&A chatbot over a knowledge base
This section explains why an enterprise Q&A chatbot needs focused content from a knowledge base rather than a prompt containing all company information.
- Single-Text Chatbot Design: Describe how the basic Q&A chatbot answers questions about one text: the prompt supplies the text and question, instructs the chatbot to answer only from that text, and a session retains conversation history to refine answers.
Why it matters: This simple design provides a contrast for the larger knowledge-base problem. - Focused Knowledge-Base Context: Explain why an enterprise Q&A chatbot should retrieve only content relevant to a question instead of putting all company content in the prompt. Describe how this lets a user ask naturally without supplying the background documents each time.
Why it matters: Choosing focused context makes Q&A over many company sources feasible and more useful. - Standard Chatbot Access Limits: Distinguish standard ChatGPT's inability to connect directly to local company data from the limited document-upload option offered by a custom GPT, and state why more control calls for a different approach.
Why it matters: It clarifies why a simple document-upload option is not the same as a controllable enterprise Q&A design.
Key terms: prompt, session, conversation history, knowledge base, context window, retrieve, context, ChatGPT, My GPTs, RAG design pattern
6.1.3 The RAG design pattern
Explains how a RAG chatbot prepares documents for semantic retrieval, then uses retrieved chunks as context for an LLM-generated answer.
- RAG stages and roles: Explain what retrieval, augmentation, and generation mean in this Q&A design, and distinguish content ingestion from question answering.
Why it matters: The two-stage picture makes it clear what must happen before a chatbot can answer questions over your content. - Chunk and embedding preparation: Trace how source documents become searchable chunks and explain why the text is split and embedded.
Why it matters: Preparation determines whether a later question can find a focused, relevant passage even when its wording differs. - Question-time retrieval and generation: Trace a user question through embedding, similarity search, prompt construction, and LLM answer generation.
Why it matters: This sequence identifies where relevant evidence enters the answer rather than treating the LLM as the search engine. - Single-document Q&A contrast: Distinguish this vector-store RAG chatbot from the earlier single-document Q&A example.
Why it matters: The comparison shows what retrieval adds without changing the basic question-plus-context approach to generation.
Key terms: Retrieval Augmented Generation (RAG), content ingestion stage (indexing), question-answering stage (Q&A; retrieval and generation), context, content ingestion, chunks, embedding, vector database, semantic search, retriever
6.2 Vector stores
This introduction previews an explanation of what vector stores are, their purpose, what they offer, and examples.
6.2.1 What’s a vector store?
Defines vector stores and explains how stored embeddings support semantic similarity searches.
- Vector stores and embeddings: Explain what a vector store holds and why embeddings are useful for indexing content.
Why it matters: This explains what the store contains when an application searches its content. - Similarity search by vector distance: Describe how a vector store finds content similar to a query and what that similarity means.
Why it matters: It connects a numerical search result to finding chunks with related meanings.
Key terms: vector store, high-dimensional vectors, embeddings, indexes, text chunks, similarity searches, query embedding, stored chunks, closest vectors, semantic similarity
6.2.2 How do vector stores work?
This section introduces vector similarity search, contrasts dense and sparse vectors, and traces vector stores from image search to text-based search.
- Vector distance and similarity search: Explain that vector search compares vectors using distance calculations, and that KNN and the more scalable ANN search use such calculations to find similar vectors.
Why it matters: This gives you the basic idea behind finding similar items in a vector store. - Dense and sparse search vectors: Distinguish dense vectors, whose dimensions are mostly nonzero and were used for image-embedding search, from sparse vectors, whose values are mostly zero and were used for exact-word lexical search.
Why it matters: The distinction clarifies how the vector search described here differs from earlier exact-word search. - Expansion of vector store uses: Describe how early vector stores such as Milvus supported image-embedding search, then expanded to uses such as product recommendations and, with LLMs, text-based semantic similarity search.
Why it matters: This explains why technology first used for image embeddings appears in text-based LLM applications.
Key terms: vector distance, Euclidean distance, cosine distance, Hamming distance, k-Nearest Neighbors (KNN), Approximate Nearest Neighbor (ANN), dense vector search, sparse vectors, image embeddings, lexical search
6.2.3 Vector libraries vs. vector databases
The section contrasts early vector libraries with vector databases and explains why their terminology has converged.
- Updates and ingestion queries: Distinguish early vector libraries from vector databases in their ability to update data and run similarity searches while importing new data.
Why it matters: These differences explain why vector databases better suit changing, actively used data. - Text and embedding linkage: Explain how storing original content separately from embeddings complicated early vector libraries, and how vector databases simplified that relationship.
Why it matters: Keeping content and embeddings together removes linking work from the application. - Converging vector store terminology: Describe how vector databases gained database-like capabilities, existing databases added vector support, and the section consequently uses three vector-store terms.
Why it matters: This explains why the book uses different names for increasingly similar storage systems.
Key terms: vector libraries, vector databases, immutable data structures, CRUD, querying during import, embeddings, original data, unique identifier, metadata, vector store
6.2.4 Most popular vector stores
This subsection frames Table 6.1 as a provisional overview of popular vector stores rather than a definitive comparison.
- Limits of vector store comparisons: Explain why a comparison of popular vector stores is only a starting point for exploring the market.
Why it matters: It helps you treat the table as an orientation aid, not a lasting or exhaustive comparison.
Key terms: vector stores, rapid evolution and convergence
6.2.5 Storing text and performing a semantic search using Chroma
This section shows how to store text chunks in Chroma, retrieve semantically similar chunks, and distinguish retrieval from answer generation.
- Chroma collection and ingestion: Describe how the example creates an in-memory Chroma client and a collection, then stores manually split Paestum text chunks with IDs and source metadata. Explain that Chroma generates embeddings from the supplied text.
Why it matters: Retrieval works because the text has already been stored and embedded. - Semantic search and distance: Explain how querying the populated collection retrieves the closest text chunks and how embedding distances indicate which chunks are more relevant to the question.
Why it matters: The returned chunks and distances show which stored passages Chroma considers relevant. - Retrieval versus answer generation: Distinguish Chroma’s retrieval of relevant text chunks from an LLM’s generation of a formulated answer using the question and retrieved chunks.
Why it matters: Separating retrieval from generation clarifies what each component contributes to a Q&A chatbot. - Chroma client deployment options: Describe the client options mentioned besides the example’s in-memory client: a local on-disk instance or an HTTP client connecting to Chroma on the same or another computer.
Why it matters: The notebook’s temporary storage is only one way to run Chroma.
Key terms: in-memory client, collection, chunks, documents, embeddings, metadata, semantic search, distance, closest result, vector database
6.3 Implementing RAG from scratch
Introduces a RAG chatbot that retrieves content from ChromaDB so a user can ask a question without supplying the source text in the prompt.
- Retrieval-backed chatbot answer flow: Trace how the proposed chatbot answers the Paestum question: the user asks only the question, the chatbot retrieves content from ChromaDB, and GPT-5-nano receives that content along with the original question.
Why it matters: This is the central RAG flow that the subsequent implementation puts into code.
Key terms: RAG, ChromaDB, GPT-5-nano, original question
6.3.1 Retrieving content from the vector database
This subsection wraps an existing vector-store semantic search in a reusable function and checks that it retrieves the expected text.
- Reusable content retrieval: Describe how the section reuses semantic-search code in a function and checks its result against a previously asked question.
Why it matters: Retrieval supplies the relevant content that the chatbot can use when answering a question.
Key terms: semantic search, vector store, retrieved chunk
6.3.2 Invoking the LLM
This section shows how to send a retrieved-context Q&A prompt to an LLM and use prompt instructions to reduce unsupported answers.
- Question and context prompt: Describe how the user's question and context retrieved from the vector database are combined in a prompt and submitted to the LLM.
Why it matters: This connects retrieval to answer generation in the Q&A flow. - Safer Q&A prompt instructions: Explain how the recommended RAG prompt aims to mitigate hallucinations, including what it tells the LLM to do when the retrieved context does not support an answer.
Why it matters: An explicit instruction to admit uncertainty helps avoid answers unsupported by the stored knowledge.
Key terms: prompt, user’s question, context, vector database, LLM, hallucinations, retrieved context, hallucination-safe RAG prompt
6.3.3 Building the chatbot
This section presents a basic chatbot that answers questions from text imported into a vector database and states the limits of its answers.
- Chatbot answer boundaries: Describe how the basic chatbot answers questions using text imported into the vector database and the section’s claim about information absent from that database.
Why it matters: This identifies what the chatbot uses to answer questions and the answer boundary the section claims for it.
Key terms: chatbot, vector database, hallucinate
6.3.4 Recap of RAG terminology
This subsection points readers to a RAG glossary and previews a LangChain reimplementation.
6.4 Summary
Summarizes why RAG scales Q&A beyond a single supporting document, how its retrieval pipeline works, and what affects retrieval quality.
Where this is used later
- Generating queries for a graph database (chapter 10) · the idea of semantic search in retrieval
- Feeding the chat history to the RAG chain (chapter 7) · Builds on the idea of feeding chat history into a Q&A interaction.
- LangChain object model for Q&A chatbots (chapter 7) · Provides the distinction between indexing and retrieval-and-generation stages.
- Splitting and storing the documents (chapter 7) · Implements document splitting and storage.
- Rewrite-Retrieve-Read (chapter 9) · Assumes the basic RAG pattern of retrieval followed by answer generation.
- Completing the RAG chain setup (chapter 7) · Combines retrieval and generation in a RAG chain.
- Q&A across stored documents (chapter 7) · Extends Q&A across stored documents.
- Embedding hypothetical questions (chapter 8) · Vector search compares a query embedding with stored vectors.
Concepts explained in this chapter
- conversational aiA chatbot is a conversational software interface that maintains context across multiple exchanges for dialogue and interactive question answering.
- retrieval-augmented generationRetrieval-augmented generation is a technique that retrieves relevant external information and supplies it to a language model as context for response generation.
- vector databaseA vector store is a database specialized for storing embeddings and associated documents and retrieving them by semantic similarity.
- similarity searchRetrieval is the process of finding relevant stored documents from an indexed collection in response to a query.
- chat APIA chat API is an interface for exchanging conversational messages with a language model and receiving responses.
- vectorizationAn embedding model is a model that maps text or other data to numerical vectors representing semantic or contextual properties.
- hallucinationA hallucination is an incorrect, misleading, or fabricated response generated by a language model.
- promptA system prompt is an instruction message supplied to a language model to establish high-level behavior or constraints.
- embeddingA vector type is a database data type that represents numerical vector values as stored fields.
- DocumentA document is a LangChain container holding content together with metadata such as source, page number, author, or timestamp.
- contextPrompt context is background text or information supplied alongside a request to help a language model interpret it and formulate an answer.
- dialogueA chat message history is an ordered collection of user and language-model messages representing a conversation.
- prompt securityA safe prompt is an instruction that restricts a language model to supplied information and directs it to acknowledge when an answer is unavailable.
- q&a engineA Q&A engine is a software system that answers user questions using information retrieved from a knowledge base.
- stateful sessionA stateful session is an interaction session that retains prior exchanges and makes them available to later requests.
- Chroma DBChroma is a vector database that stores text, embeddings, and metadata and supports similarity-based retrieval.
- pipelineA multi-step workflow is a process composed of several ordered or coordinated operations that collectively accomplish a task.
- question-answering stageA question-answering stage is the retrieval-and-generation phase that obtains relevant content for a user question and generates an answer from it.
- text chunkA text splitter is a component that divides large text into smaller chunks or document objects.
- collectionA collection is a Chroma container for documents, embeddings, metadata, and identifiers.
- context windowA context window is the maximum amount of input and conversational context a language model can process in one interaction.
- knowledge baseAn internal knowledge base is information encoded in a language model's learned parameters and available without consulting external data sources.
- lexical searchLexical search retrieves content primarily by matching literal words or terms.
- GPT-5-nanoGPT-5-nano is a small GPT-5 model variant intended for fast, lower-cost language processing.
- memoryMemory is stored information that an agent or application can retain and use across steps or interactions.
- templateA prompt template is a reusable structure that combines user input, context, and optionally examples into a model request.
← Chapter 5: Agentic workflows with LangGraph
Chapter 7: Q&A chatbots with LangChain and LangSmith →
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.