Chapter 3: Advanced vector retrieval strategies
This chapter introduction motivates improving a basic RAG retriever with query rewriting and advanced document-embedding strategies to retrieve more relevant, complete context for answer generation.
After this chapter you will be able to
- Query rewriting for retrieval. Explain how query rewriting bridges a mismatch between a user's wording and the terminology or context in relevant documents, improving retrieval accuracy and recall. Describe step-back prompting as rewriting a detailed query into a broader, more general question before vector search.
- Advanced document embedding strategies. Describe hypothetical question embeddings and parent document embeddings as alternatives to embedding exactly the text that will be returned. Distinguish matching a user query to questions a document can answer from matching it to child chunks and then returning the associated full parent document.
- Step-back query transformation. Explain how step-back prompting rewrites a specific user question into a broader, higher-level question before vector retrieval. Describe that the broader query can cover more relevant information, reducing search complexity and improving retrieval precision, recall, and ultimately answer accuracy.
- LLM rewriting execution. Describe the procedure for generating a step-back question: send an LLM a system prompt containing rewriting instructions together with the user's question, then use the broader output as the rewritten query. State that an existing LLM can perform this task through prompting without training or finetuning a separate model.
- Parent-child retrieval structure. Explain how the parent document retriever represents a PDF as parent documents and child documents, embedding the child chunks for retrieval while retaining their larger parent text for context-rich responses.
- Structure-aware document preprocessing. Describe the preprocessing sequence used for the paper: split the PDF text into structural sections first, inspect their token lengths, and split overly long sections into parent documents of at most 2,000 characters.
- Parent document retrieval query. Describe how the retrieval Cypher statement finds relevant parent documents by performing vector search over child nodes using a question embedding, mapping results to parents, and enforcing a final limit of k unique documents.
- Oversampling for parent deduplication. Explain why the initial vector search retrieves k * 4 child documents: several highly similar children can belong to one parent, so the larger candidate pool is a safety buffer for producing k unique parent documents after deduplication.
- Complete advanced RAG pipeline. Trace how the rag_pipeline function turns an input question into a final answer by creating a step-back prompt, retrieving related parent documents with that rewritten prompt, and giving the retrieved documents plus the original question to an LLM.
What it builds on
- Embedding model (chapter 2) · Text embeddings represent the semantic meaning of a query and document.
- Performing vector search (chapter 2) · Vector search uses a query embedding to retrieve semantically similar text.
- Text chunking (chapter 2) · Text chunking provides the basis for splitting a parent document into child chunks.
- Performing hybrid search (chapter 2) · Hybrid search combines full-text and vector retrieval.
- The retriever (chapter 2) · A retriever obtains relevant context for a RAG application.
- The text corpus (chapter 2) · Using a text corpus as the source data for a RAG application.
- Database with vector similarity search function (chapter 2) · Using a database with vector similarity search.
- Generating an answer using an LLM (chapter 2) · The pipeline uses the established pattern of passing a question and relevant documents to an LLM for answer generation.
Section by section
Introduction
This chapter introduction motivates improving a basic RAG retriever with query rewriting and advanced document-embedding strategies to retrieve more relevant, complete context for answer generation.
- Query rewriting for retrieval: Explain how query rewriting bridges a mismatch between a user's wording and the terminology or context in relevant documents, improving retrieval accuracy and recall. Describe step-back prompting as rewriting a detailed query into a broader, more general question before vector search.
Why it matters: A better retrieval query can find relevant evidence that the original wording would otherwise miss. - Advanced document embedding strategies: Describe hypothetical question embeddings and parent document embeddings as alternatives to embedding exactly the text that will be returned. Distinguish matching a user query to questions a document can answer from matching it to child chunks and then returning the associated full parent document.
Why it matters: These strategies can make matching more precise while preserving the context needed for a complete LLM answer. - Retrieval improvement tradeoffs: State why embedding an entire long document can blur distinct ideas through averaging, and why smaller chunks improve query-specific matching while parent retrieval preserves full context. Identify finetuning, reranking, metadata filtering, and hybrid retrieval as additional retrieval-quality techniques mentioned without implementation coverage in this chapter.
Why it matters: Understanding the tradeoff between precise matching and sufficient context helps explain why parent document retrieval is useful.
Key terms: query rewriting, step-back prompting, retrieval accuracy, recall, hypothetical question–embedding strategy, parent document–embedding strategy, parent document, child chunks, nearest neighbors, averaging
3.1 Step-back prompting
This section shows how step-back prompting uses an LLM to rewrite detailed questions into broader queries that can improve vector retrieval.
- Step-back query transformation: Explain how step-back prompting rewrites a specific user question into a broader, higher-level question before vector retrieval. Describe that the broader query can cover more relevant information, reducing search complexity and improving retrieval precision, recall, and ultimately answer accuracy.
Why it matters: Broadening an overly specific query can help vector search retrieve the facts needed to answer it accurately. - LLM rewriting execution: Describe the procedure for generating a step-back question: send an LLM a system prompt containing rewriting instructions together with the user's question, then use the broader output as the rewritten query. State that an existing LLM can perform this task through prompting without training or finetuning a separate model.
Why it matters: This makes query rewriting a practical retrieval improvement that can be added by prompting an existing LLM. - Zero-shot and few-shot prompting: Distinguish zero-shot prompting from few-shot prompting for step-back rewriting. Explain that zero-shot gives only the rewriting instruction, whereas few-shot adds a small number of example transformations—typically two to five—to guide the desired paraphrasing and make output more consistent and reliable.
Why it matters: Examples in a prompt can make the LLM's query rewrites better match the intended broader-question format.
Key terms: step-back prompting, query rewriting, vector retrieval, step-back question, system prompt, LLM, zero-shot prompting, few-shot prompting, examples, paraphrasing behavior
3.2 Parent document retriever
This section prepares a parent-document retrieval structure by hierarchically chunking a PDF, embedding child chunks, and importing the resulting graph into Neo4j.
- Parent-child retrieval structure: Explain how the parent document retriever represents a PDF as parent documents and child documents, embedding the child chunks for retrieval while retaining their larger parent text for context-rich responses.
Why it matters: This structure improves query matching with small chunks without losing the broader context needed by the LLM. - Structure-aware document preprocessing: Describe the preprocessing sequence used for the paper: split the PDF text into structural sections first, inspect their token lengths, and split overly long sections into parent documents of at most 2,000 characters.
Why it matters: Starting with meaningful document structure preserves coherent ideas while token and size checks prevent overly large LLM context inputs. - Graph import and child indexing: Describe how the graph import creates a PDF node, uniquely identified Parent nodes connected by HAS_PARENT relationships, and Child nodes connected by HAS_CHILD relationships, then creates a vector index on the Child embedding property.
Why it matters: The graph links let retrieval move from a matched child chunk to its parent context, while the child vector index enables efficient embedding comparison.
Key terms: parent document retriever, parent documents, child documents, embedding, PDF node, Parent node, split_text_by_titles, regular expression, sections, tiktoken
3.2.1 Retrieving parent document strategy data
This section implements parent-document retrieval by searching child-node vectors, deduplicating their parents, and returning a final set of relevant documents.
- Parent document retrieval query: Describe how the retrieval Cypher statement finds relevant parent documents by performing vector search over child nodes using a question embedding, mapping results to parents, and enforcing a final limit of k unique documents.
Why it matters: This query turns matches on small child chunks into distinct parent documents that can provide fuller context to the RAG system. - Oversampling for parent deduplication: Explain why the initial vector search retrieves k * 4 child documents: several highly similar children can belong to one parent, so the larger candidate pool is a safety buffer for producing k unique parent documents after deduplication.
Why it matters: Oversampling prevents repeated matches from one parent document from leaving too few distinct documents in the final retrieval results. - Parent retrieval function flow: Trace the parent_retrieval function: it first generates a text embedding for the question, then executes the retrieval Cypher statement to return the most relevant documents from the database.
Why it matters: The function connects a natural-language question to the database query that supplies relevant parent-document context.
Key terms: Cypher statement, vector-based search, child nodes, question embedding, parent documents, final k limit, k * 4, initial vector search, deduplicate, unique parent documents
3.3 Complete RAG pipeline
This section combines step-back prompting, parent document retrieval, and LLM answer generation into a complete RAG pipeline and suggests testing its impact.
- Complete advanced RAG pipeline: Trace how the rag_pipeline function turns an input question into a final answer by creating a step-back prompt, retrieving related parent documents with that rewritten prompt, and giving the retrieved documents plus the original question to an LLM.
Why it matters: This shows how the chapter's advanced retrieval techniques work together in one answer-producing workflow. - Step-back prompting comparison: Describe how to evaluate the completed pipeline by asking questions about Einstein's life in the PDF and comparing its results with a version in which the step-back prompting step is removed.
Why it matters: Comparing the two versions lets you assess whether query rewriting actually improves results for the questions you test.
Key terms: rag_pipeline, step-back prompt, parent document retrieval, original question, LLM, final answer, step-back prompting, Einstein's life, PDF
3.4 Summary
This summary recaps advanced vector-retrieval strategies that improve query-document alignment and retrieve more relevant context for RAG answers.
Where this is used later
- Using few-shot examples for in-context learning (chapter 4) · Few-shot examples are presented as a useful practice for guiding LLM query-language generation.
- Tying it all together (chapter 5) · Tying agentic RAG components together requires understanding how retrieval and answer generation form an end-to-end pipeline.
- (chapter intro) (chapter 4) · The existing RAG pipeline and its retrieval role.
- The basics of query language generation (chapter 4) · The idea of a pipeline that retrieves input and passes prepared context to an LLM.
- Where query language generation fits in the RAG pipeline (chapter 4) · A complete RAG pipeline that can use retrieval results to generate answers.
- Summary (chapter 4) · The idea of a complete RAG pipeline with retrieval as an input to answer generation.
Concepts explained in this chapter
- ParentDocumentRetrieverParent document retrieval is a strategy that embeds or matches smaller child segments while returning their enclosing parent document for broader context.
- step-back promptingStep-back prompting is a prompting technique that reformulates a specific question into a more general question or perspective before retrieval or generation.
- hypothetical question embeddingHypothetical question embedding is a retrieval strategy that embeds questions a document could answer and matches user queries against those question embeddings.
- query rewritingQuery rewriting is the transformation of a user query into a formulation that better matches the language and context of target documents.
- text chunkText chunking is the process of splitting source text into smaller pieces for downstream processing and retrieval.
- parent documentA parent document is a document segment that contains smaller child documents and provides a larger context unit for retrieval.
- child documentA child document is a smaller text segment derived from a parent document and associated with an embedding for retrieval.
- tiktokentiktoken is a tokenization library developed by OpenAI for encoding text and counting language-model tokens.
- retrieval-augmented generationA retrieval-augmented generation pipeline combines information retrieval with language-model generation to produce responses grounded in retrieved information.
- hybrid searchHybrid search is an information-retrieval technique that combines multiple retrieval methods, such as vector similarity search and full-text search.
← Chapter 2: Vector similarity search and hybrid search
Chapter 4: Generating Cypher queries from natural language questions →
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.