Chapter 4: Talking with your documents

Introduction

The introduction explains why giving an LLM relevant document content when a question is asked can improve answers beyond what its training alone supports.

• Relevant documents for grounded answers: Explain that retrieval-augmented generation (RAG) provides relevant information to an LLM as a question is asked, helping it answer questions its training has not prepared it for.
  Why it matters: Relevant source material gives the LLM a better basis for answering questions its training alone cannot cover.

Key terms: retrieval-augmented generation (RAG), LLM, hallucinations

4.1 Understanding RAG

Explains why retrieval-augmented generation (RAG) selects relevant document chunks for a prompt instead of stuffing entire rulebooks into it.

• Limits of prompt stuffing: Explain why putting every game rulebook into a prompt is impractical and how RAG addresses that problem.
  Why it matters: This explains why retrieving a little relevant context is more practical than supplying every rulebook with every question.
• Document retrieval and prompt context: Trace how document chunks and a question pass through a typical RAG system to supply context for a prompt.
  Why it matters: This is the basic path by which stored rules become relevant context for an answer.
• Embedding and vector store abstractions: Explain which RAG details the developer does not need to calculate directly and what handles them.
  Why it matters: It clarifies the level of embedding and search detail needed to work with RAG in Spring AI.

Key terms: RAG, prompt stuffing, context window, document chunks, document loader, vector store, embeddings, prompt as context, embedding models, vector stores

4.2 Setting up a vector store

The section explains how vector search finds similar content and how to run Qdrant as a vector store for the Board Game Buddy application.

• Vector Similarity Search: Explain how vector search identifies content similar to a query and what the developer does not need to calculate manually.
  Why it matters: This explains what the vector store contributes when the application searches its documents.
• Vector Store Choice: Describe what factors guide the choice of a Spring AI-supported vector store and why the example uses Qdrant.
  Why it matters: It separates the practical choice of a store from the application code that uses one.
• Qdrant Compose Configuration: Describe where the Docker Compose configuration goes and what the two exposed Qdrant ports are for.
  Why it matters: The port mapping lets the application reach Qdrant and gives developers an optional way to inspect it.
• Qdrant Startup Options: Distinguish manually starting Qdrant with Docker Compose from having Spring Boot start it, including the roles of the two dependencies described.
  Why it matters: Spring Boot can manage both the local container lifecycle and the application's connection to it.

Key terms: vector search, cosine similarity, cosine distance, vector store, Spring AI, Qdrant, Docker Compose, compose.yaml, port 6334, port 6333

4.3 Loading documents

The section explains the basic document-loading flow for a vector store and why Board Game Buddy needs a pipeline that can load multiple documents over time.

• Basic document loading flow: Trace how Spring AI reads a file, splits its content into chunks, and saves those chunks to a vector store.
  Why it matters: This flow is the foundation for loading material that a RAG application can use.
• Need for a loading pipeline: Explain why Board Game Buddy needs a document-loading pipeline instead of loading one declared Resource at application startup.
  Why it matters: The pipeline accommodates a growing game library without requiring an application restart for each new rules document.

Key terms: document reader, text splitter, vector store client, TextReader, TokenTextSplitter, Document, document-loading pipeline, Resource, Board Game Buddy, vector store

4.3.1 Initializing the loader project

The section sets up a Spring Boot document-loader project, explaining its dependencies, build configuration, and properties for writing documents to Qdrant.

• Loader pipeline library roles: Describe how the selected Spring libraries support a loader that watches for files and writes documents to a vector store.
  Why it matters: These roles explain why the loader project needs this particular mix of dependencies.
• Embedding model compatibility: Explain why the loader and game rules application must use the same or compatible embedding API.
  Why it matters: The application that uses the stored documents needs embeddings compatible with those the loader created.
• File supplier build setup: Describe the extra build configuration needed for the Spring Function Catalog file supplier.
  Why it matters: Selecting Initializr dependencies alone will not make the file supplier available to the loader.
• Loader application properties: Explain the configuration for the loader’s port, Qdrant collection, and OpenAI API key.
  Why it matters: These settings let the loader start alongside the game rules application and write to the collection that application will use.

Key terms: fileSupplier, custom consumer, Spring Cloud Function, Spring AI Tika Document Reader, Spring AI Qdrant, embedding API, embedding models, game rules application, Spring Function Catalog, file supplier dependency

4.3.2 Defining the loader pipeline

This section defines a Spring Cloud Function pipeline that selects files, turns them into document chunks, and writes those chunks to a vector store.

• Composed function definition: Explain how spring.cloud.function.definition defines the loader pipeline and how its function names relate to Spring beans.
  Why it matters: The property makes the pipeline’s order visible in one place.
• File supplier selection: Describe how fileSupplier obtains files and how its directory and filename filter are configured.
  Why it matters: File selection determines which inputs enter the document-loading pipeline.
• Document chunk loading sequence: Trace what documentReader, splitter, and vectorStoreConsumer do to a file after it is picked up.
  Why it matters: This sequence explains how an incoming file becomes content stored for later use.

Key terms: spring.cloud.function.definition, composed function, Spring application context, beans, fileSupplier, file.supplier.directory, file.supplier.filename-regex, documentReader, Document, splitter

4.3.3 Creating the pipeline components

This section implements a document-loading pipeline that reads game rules, splits them into chunks, adds game-title metadata, and writes the chunks to a vector store.

• Functional pipeline and stream: Explain how the pipeline components connect and why they pass a Flux through the document-loading flow.
  Why it matters: This explains how files can move through the loader as they become available.
• Flexible document reader choice: Justify the choice of TikaDocumentReader over the PDF readers for this rules loader.
  Why it matters: The reader choice lets the loader accept rules beyond a single PDF format.
• File bytes to documents: Trace how the documentReader function turns the file supplier’s bytes into documents for the next component.
  Why it matters: It connects the file supplier’s output to the document-processing stages.
• Token-based document chunks: Explain why the loader splits Tika-read documents and describe how TokenTextSplitter determines chunk boundaries.
  Why it matters: Chunking makes large rulebooks usable as smaller pieces of prompt context.
• Game title metadata: Explain how and why titleDeterminer adds a normalized game title to the document chunks.
  Why it matters: Title metadata helps prevent answers based on the wrong game’s rules.
• Vector store consumer subscription: Describe how the final consumer writes arriving chunk lists and why it subscribes to the Flux.
  Why it matters: Without the final subscription, the assembled loader would not write its chunks.

Key terms: Function, Consumer, Flux, @Bean, composite function, TikaDocumentReader, PagePdfDocumentReader, ParagraphPdfDocumentReader, table of contents metadata, documentReader

4.3.4 Running the pipeline

This section starts the document-loading pipeline and shows how to confirm that PDF rulebooks were written to Qdrant.

• Starting the composed pipeline: Explain how the application starts the document-loading pipeline using an ApplicationRunner bean and the composed function from FunctionCatalog.
  Why it matters: Defining pipeline components is not enough; the application must start the composed function.
• Verifying document loading: Describe how to test the running loader with a PDF rulebook and verify its output using the logs and Qdrant’s API.
  Why it matters: These checks show whether the loader actually placed searchable document chunks in the vector store.

Key terms: spring.cloud.function.definition, ApplicationRunner, FunctionCatalog, lookup(), run(), /tmp/dropoff, Qdrant, /collections, points, exact

4.4 Implementing RAG

Introduces the RAG flow for Board Game Buddy and identifies the vector store dependency needed before implementing it.

• RAG retrieval and context: Explain that RAG first searches a vector store for documents similar to the user's question, then includes those documents in the prompt as context.
  Why it matters: This sequence is the basis for adding document context to Board Game Buddy's answers.
• Matching Qdrant dependency: State that Board Game Buddy needs the Qdrant vector store starter dependency because the document loader used Qdrant.
  Why it matters: Board Game Buddy needs the dependency for the vector store chosen when its documents were loaded.

Key terms: RAG, vector store, prompt, context, Qdrant, vector store starter dependency, Board Game Buddy

4.4.1 Searching for similar documents

This section changes GameRulesService from loading rules at a fixed location to retrieving relevant rule chunks from a vector store for a specific game.

• Question-Based Game Rules Search: Explain why GameRulesService searches document chunks instead of loading a fixed rulebook, and describe how its SearchRequest limits the search to the game being asked about.
  Why it matters: The question finds relevant chunks, while the game-title filter keeps the search focused on the right game.
• Top-K Context Tradeoff: Explain what Top-K controls in a similarity search and why choosing its value involves a tradeoff.
  Why it matters: Top-K balances enough rule context against token cost and limits.
• Similarity Threshold Tradeoff: Explain what the similarity threshold filters out and why setting it too high can hurt an answer.
  Why it matters: A stricter match can improve relevance only if it still leaves enough rules to answer from.
• Search Results as Rule Context: Trace what getRulesFor() returns after the vector store search and how that text is used in the prompt.
  Why it matters: This turns retrieved chunks into the rule text supplied as prompt context.

Key terms: GameRulesService, VectorStore, SearchRequest, gameTitle, normalized game title, FilterExpressionBuilder, Top-K, withTopK(), prompt context, tokens

4.4.2 Updating the service

Updates the board-game question service to use retrieved rules, tightens the prompt to discourage unsupported answers, and shows how to try the result.

• Question-aware rules service call: Describe why askQuestion() must pass both the game name and the question to GameRulesService.getRulesFor(), while continuing to put the returned rules text into the prompt.
  Why it matters: This identifies the small service change needed to connect retrieval to the existing question-and-answer prompt.
• Rules-grounded prompt response: Explain why the prompt is made stricter and what it instructs the LLM to say when the supplied rules do not contain the answer.
  Why it matters: The stricter instruction aims to prevent confident answers that the supplied rules do not support.
• Retrieval test and model variation: Describe how the example tries questions with and without loaded game rules, and why the response may not exactly follow the prompt's instruction.
  Why it matters: The example shows both how to try the change and why prompt instructions should not be mistaken for guaranteed model behavior.

Key terms: SpringAiBoardGameService, GameRulesService, getRulesFor(), askQuestion(), {rules}, systemPromptTemplate.st, hallucinations, I don’t know, vector store, /ask endpoint

4.5 Implementing RAG with an advisor

This section shows how QuestionAnswerAdvisor handles RAG retrieval and prompt context, then configures it as a default ChatClient advisor with a per-request game-title filter.

• QuestionAnswerAdvisor RAG behavior: Explain how QuestionAnswerAdvisor replaces the manual RAG work in askQuestion(): it searches a VectorStore for documents similar to the question and adds the retrieved documents as prompt context before the request reaches the LLM.
  Why it matters: It lets you explain where retrieval and prompt augmentation happen when the application no longer does them explicitly.
• Default advisor and request filter: Describe how to make QuestionAnswerAdvisor a default on the ChatClient and still restrict retrieval to the requested game. Explain why askQuestion() supplies the game-title filter at request time rather than when the ChatClient bean is created.
  Why it matters: This separates reusable advisor configuration from the game-specific information available only when a question arrives.

Key terms: QuestionAnswerAdvisor, ChatClient, VectorStore, similarity search, defaultAdvisors(), advisors(), FILTER_EXPRESSION, game title filter, SearchRequest

4.6 Applying modular RAG

This section introduces RetrievalAugmentationAdvisor as a configurable alternative to QuestionAnswerAdvisor and shows how to use it with a vector-store retriever and prompt-time filters.

• Pluggable document retrieval: Distinguish how RetrievalAugmentationAdvisor and QuestionAnswerAdvisor choose where to retrieve documents. Explain why RetrievalAugmentationAdvisor requires a document retriever and why this example uses VectorStoreDocumentRetriever.
  Why it matters: Choosing the retriever is what makes this advisor adaptable to document sources beyond a vector store.
• Advisor setup and filtering: Describe how the example makes RetrievalAugmentationAdvisor available to ChatClient and applies a vector-store filter when building a prompt.
  Why it matters: A default advisor does not eliminate prompt-specific filtering, and the filter constant must match the retriever being used.

Key terms: RetrievalAugmentationAdvisor, QuestionAnswerAdvisor, DocumentRetriever, VectorStoreDocumentRetriever, RAG dependency, default advisor, ChatClient, advisors(), FILTER_EXPRESSION

4.6.1 Rewriting the user’s query

This subsection explains why a focused query may improve document retrieval and how to configure Spring AI to rewrite a user’s query before vector-store search.

• Focused queries for retrieval: Explain why shortening and clarifying a user’s question may help retrieve more relevant documents from a vector store.
  Why it matters: The rewrite is meant to improve which documents the search retrieves, not merely to make the question shorter.
• Rewrite transformer configuration: Describe how RewriteQueryTransformer uses an LLM through a ChatClient to rewrite a query, and how it is supplied to RetrievalAugmentationAdvisor.
  Why it matters: This connects the LLM-based rewrite to the advisor that prepares a query for retrieval.

Key terms: query, vector store, RewriteQueryTransformer, ChatClient.Builder, chatClientBuilder(), RetrievalAugmentationAdvisor, queryTransformers()

4.6.2 Translating user queries

This section explains how translating user queries into the language of stored documents can improve retrieval, and how to configure that translation alongside query rewriting.

• Language matching for retrieval: Explain why translating a user's question into the language of the vector-store documents can improve search results without storing multiple translations of those documents.
  Why it matters: Users can ask in their own language while retrieval searches documents in the language in which they are stored.
• Translation transformer setup and order: Describe how the section configures TranslationQueryTransformer for English rules and how it operates alongside RewriteQueryTransformer before vector search.
  Why it matters: Translation and rewriting serve different purposes, and the example shows how they prepare a question for retrieval together.

Key terms: TranslationQueryTransformer, vector store, target language, RetrievalAugmentationAdvisor, queryTransformers(), RewriteQueryTransformer, ChatClient .Builder, chatClientBuilder()

4.6.3 Expanding user queries

This section explains how MultiQueryExpander broadens vector retrieval with alternative phrasings, how to configure it, and the tradeoff between explicit RAG and advisor-based RAG.

• Expanded queries for retrieval: Explain how MultiQueryExpander differs from RewriteQueryTransformer and why multiple phrasings of a question can help vector search find relevant documents.
  Why it matters: Alternative phrasings can improve retrieval when a relevant document uses different wording from the user.
• MultiQueryExpander advisor configuration: Describe how to supply MultiQueryExpander to RetrievalAugmentationAdvisor and how its query-count and original-query settings affect the resulting list.
  Why it matters: The settings determine both how many search phrasings are produced and whether the user’s wording remains among them.
• Explicit versus advisor RAG: Explain the section’s tradeoff between implementing RAG explicitly and letting an advisor perform it.
  Why it matters: It helps you choose between simpler advisor-based code and greater control over retrieval.

Key terms: MultiQueryExpander, RewriteQueryTransformer, vector search, ChatClient.Builder, RetrievalAugmentationAdvisor, queryExpander(), numberOfQueries(), includeOriginal, GameRulesService, QuestionAnswerAdvisor

4.7 Summary