Chapter 7: HyDE: Imagining the answer before you search
Introduces query-document mismatch and HyDE’s idea of searching with an imagined answer while grounding the final response in retrieved documents.
After this chapter you will be able to
- Query-document mismatch failures. Explain how a knowledge base can contain the answer but retrieval still fail, either by omitting the answer from context or ranking it below less useful passages.
- Hypothetical document search. Describe HyDE’s central idea: have an LLM generate a hypothetical answer-like document and use it in place of the sparse question to search by document-to-document similarity, without relevance labels for retriever fine-tuning.
- Imagined passage versus evidence. Distinguish the hypothetical document’s retrieval role from the factual grounding of the final answer: the generated passage models relevance, is discarded after retrieval, and is not used as answer evidence.
- Query–document semantic gap. Explain how a short, underspecified query and a long, technical document can be about the same thing yet end up far apart in embedding space, especially when no labeled pairs are available for domain-specific fine-tuning.
- Hypothetical answer as search text. Describe how HyDE replaces the original query with an imagined answer for retrieval, turning a query-to-document comparison into a document-to-document comparison. Explain why matching the corpus's register and vocabulary matters more here than the imagined answer being factually correct.
- Hypothetical document as query. Explain how HyDE uses an LLM to generate a passage from a short query, then uses the imagined document in place of the query for document-to-document retrieval. Distinguish this replacement from merely adding generated context to the query.
- Relevance versus factual grounding. Explain why HyDE's imagined document need not be factually accurate and how the LLM's relevance modeling differs from the retrieval step's factual grounding.
- Shared vector space search. Trace how HyDE embeds a hypothetical document and uses its vector to search real documents. Explain why the hypothetical and real documents must be embedded with the same encoder.
- Dense bottleneck and framing. Explain why HyDE can withstand a wrong name or date in its generated passage, and distinguish that from generating a passage about the wrong subject.
- Zero-shot pseudo-supervision. Explain why HyDE needs no labeled query-document pairs or fine-tuning, how the generated passage supplies a relevance signal, and how the generation prompt can be adapted to another domain.
- Domain knowledge and vocabulary limits. Explain why HyDE can help when labeled data is scarce and a query uses different language from relevant documents, and why it may fail on terms unknown to the models.
- Multilingual generation and retrieval. Trace the non-English HyDE pipeline from a query through target-language hypothetical-document generation, multilingual encoding, and corpus search. Explain what each model must be able to do.
- Clinical semantic alignment. Explain why a direct embedding of the clinician’s question may miss useful guidelines and how HyDE’s hypothetical answer improves retrieval in this example.
- Verified evidence for final answers. Distinguish the hypothetical document’s role in retrieval from the verified PubMed documents’ role in the final medical answer.
- Published HyDE vector. Explain that published HyDE averages embeddings from several hypothetical documents together with the original query embedding, rather than relying on one generated document.
- Hybrid retrieval balance. Describe how Hybrid HyDE complements a HyDE embedding with the original query embedding or fuses HyDE results with lexical search such as BM25.
- Reverse HyDE timing tradeoff. Trace how Reverse HyDE moves generation from query time to indexing time and state the resulting latency-versus-indexing tradeoff.
- Multi-sample embedding consensus. Explain how multi-sample HyDE averages the embeddings of N hypothetical documents, optionally including the query’s own embedding, and why this reduces the influence of an unusual or poor generation.
- Hybrid HyDE combination methods. Describe two ways to combine HyDE with the original query: average their vectors, or run HyDE vector search alongside lexical search and fuse the ranked results. Explain why retaining exact-match signals matters for terms such as part numbers, statute references, and drug names.
- Two-stage HyDE retrieval. Describe how HyDE first retrieves a broad candidate set and a slower, more accurate model reranks only those candidates when high precision is needed.
- Question indexed document retrieval. Trace how Reverse HyDE indexes generated questions and uses a user's question to retrieve their source documents. Explain why matching questions to questions addresses the query-document mismatch.
- Indexing versus query cost. Explain how Reverse HyDE shifts generation work from query time to indexing time and when that tradeoff is attractive.
- Learned query refinement actions. Explain how RQ-RAG makes query refinement a trained model capability rather than a runtime prompt wrapper, and describe the four actions it can select.
- Chained multi-hop refinements. Trace how RQ-RAG handles a multi-hop question by retrieving and answering sub-queries, feeding partial answers into later refinements, and selecting among candidate trajectories.
- Soft HyDE topical fingerprint. Explain why Soft HyDE can use an outline, bullets, or a short pseudo-passage instead of a polished hypothetical answer.
- Confidently wrong HyDE retrieval. Explain how a mistaken hypothetical can produce confidently wrong retrieval and what downstream checks the section recommends.
What it builds on
- RAG failure examples (chapter 1) · Introduces RAG failure examples.
- Sentence-BERT (2019): Practical sentence similarity (chapter 2) · Assumes the idea of embedding text for sentence similarity.
- The retriever (chapter 4) · Assumes familiarity with a retriever comparing a query against documents.
Section by section
Introduction
Introduces query-document mismatch and HyDE’s idea of searching with an imagined answer while grounding the final response in retrieved documents.
- Query-document mismatch failures: Explain how a knowledge base can contain the answer but retrieval still fail, either by omitting the answer from context or ranking it below less useful passages.
Why it matters: It distinguishes a retrieval problem from a missing-answer problem. - Hypothetical document search: Describe HyDE’s central idea: have an LLM generate a hypothetical answer-like document and use it in place of the sparse question to search by document-to-document similarity, without relevance labels for retriever fine-tuning.
Why it matters: It explains why HyDE searches with an answer-shaped passage rather than the user’s short question. - Imagined passage versus evidence: Distinguish the hypothetical document’s retrieval role from the factual grounding of the final answer: the generated passage models relevance, is discarded after retrieval, and is not used as answer evidence.
Why it matters: It prevents an imagined answer from being mistaken for factual evidence.
Key terms: query-document mismatch, FP1 (Missing Content), FP2 (Missed the Top Rank), HyDE, hypothetical document, document-to-document similarity, zero-shot, relevance labels, model relevance, factual grounding
7.1 The query-document mismatch problem
This section explains why short, colloquial queries can miss relevant technical documents and illustrates how HyDE narrows that mismatch by searching with an imagined answer.
- Query–document semantic gap: Explain how a short, underspecified query and a long, technical document can be about the same thing yet end up far apart in embedding space, especially when no labeled pairs are available for domain-specific fine-tuning.
Why it matters: It explains why a relevant answer can be missed even when retrieval uses semantic rather than keyword matching. - Hypothetical answer as search text: Describe how HyDE replaces the original query with an imagined answer for retrieval, turning a query-to-document comparison into a document-to-document comparison. Explain why matching the corpus's register and vocabulary matters more here than the imagined answer being factually correct.
Why it matters: The imagined answer gives retrieval text that resembles the material being searched, reducing the burden on a single query-to-document match. - Medical retrieval comparison: Trace what BM25, direct dense retrieval, and HyDE retrieve in the five-document medical example, and state what its Recall@2 result can and cannot demonstrate.
Why it matters: The side-by-side results make the retrieval failure and HyDE's advantage concrete without overstating a tiny test.
Key terms: dense retrieval, semantic gap, zero-shot, query-document mismatch, HyDE, hypothetical answer, document-only similarity, register, document encoder, BM25
7.2 Generating hypothetical documents
This section explains how HyDE turns a sparse query into an imagined document for retrieval while leaving factual grounding to real documents.
- Hypothetical document as query: Explain how HyDE uses an LLM to generate a passage from a short query, then uses the imagined document in place of the query for document-to-document retrieval. Distinguish this replacement from merely adding generated context to the query.
Why it matters: The imagined passage gives retrieval a document-like description to match rather than a sparse question. - Relevance versus factual grounding: Explain why HyDE's imagined document need not be factually accurate and how the LLM's relevance modeling differs from the retrieval step's factual grounding.
Why it matters: Separating relevance from grounding lets HyDE benefit from an LLM's language understanding without trusting its possibly false or outdated details.
Key terms: HyDE, hypothetical document, semantic gap, query embedding, encoder, relevance modeling, factual grounding, trusted corpus
7.3 Zero-shot dense retrieval without relevance labels
This section explains how HyDE turns a generated passage into a zero-shot search vector, why that vector can tolerate some factual errors, and how the approach is used and evaluated.
- Shared vector space search: Trace how HyDE embeds a hypothetical document and uses its vector to search real documents. Explain why the hypothetical and real documents must be embedded with the same encoder.
Why it matters: The shared space is what makes a generated passage usable as a search query. - Dense bottleneck and framing: Explain why HyDE can withstand a wrong name or date in its generated passage, and distinguish that from generating a passage about the wrong subject.
Why it matters: A plausible but factually imperfect passage can still help retrieval, whereas a passage aimed at the wrong topic can misdirect it. - Zero-shot pseudo-supervision: Explain why HyDE needs no labeled query-document pairs or fine-tuning, how the generated passage supplies a relevance signal, and how the generation prompt can be adapted to another domain.
Why it matters: HyDE can provide a relevance signal even where labeled retrieval examples are unavailable. - Framework retrieval integration: Describe what LangChain’s HyDE embedder does when given a query and how its returned vector is used; distinguish that wrapper from a retriever that also wraps the vector store.
Why it matters: The wrapper makes HyDE usable without changing the rest of an application’s retrieval architecture. - Zero-shot benchmark comparison: State how HyDE’s reported zero-shot retrieval performance compared with lexical, unsupervised dense, and supervised baselines in this section.
Why it matters: The comparison shows why a training-free retrieval approach is worth testing against established baselines.
Key terms: hypothetical document, unsupervised contrastively learned encoder, vector embedding, similarity search, dense bottleneck, lossy compressor, relevance pattern, semantic frame, hallucinations, zero-shot
7.4 Implementing HyDE for challenging queries
This section explains where HyDE can help with challenging queries, how to use it for non-English retrieval, and the limits and costs of doing so.
- Domain knowledge and vocabulary limits: Explain why HyDE can help when labeled data is scarce and a query uses different language from relevant documents, and why it may fail on terms unknown to the models.
Why it matters: This helps you distinguish a query HyDE might improve from one it cannot meaningfully expand. - Multilingual generation and retrieval: Trace the non-English HyDE pipeline from a query through target-language hypothetical-document generation, multilingual encoding, and corpus search. Explain what each model must be able to do.
Why it matters: The two model requirements explain how a generated answer can become a useful retrieval query in another language. - Generation quality and retrieval cost: Describe the latency and computational cost of HyDE's extra generation call, and how generator quality and low-resource language capabilities can affect retrieval quality.
Why it matters: These tradeoffs matter when deciding whether HyDE is worthwhile for a real-time or low-resource-language application.
Key terms: zero-shot, hypothetical document, vocabulary problem, multilingual generative LLM, multilingual encoder, shared vector space, query-document mismatch, latency, computational cost, low-resource languages
7.5 Case study: Medical knowledge retrieval
A medical retrieval case study shows how HyDE uses a hypothetical answer to find more relevant PubMed documents while keeping the final answer grounded in verified sources.
- Clinical semantic alignment: Explain why a direct embedding of the clinician’s question may miss useful guidelines and how HyDE’s hypothetical answer improves retrieval in this example.
Why it matters: The case illustrates how HyDE can bridge the gap between a brief clinical question and the terminology used in expert documents. - Verified evidence for final answers: Distinguish the hypothetical document’s role in retrieval from the verified PubMed documents’ role in the final medical answer.
Why it matters: In a high-stakes medical setting, a useful search prompt must not become an unverified source for the answer.
Key terms: HyDE, hypothetical answer, semantic key, domain expert simulator, PubMed vector store, hallucination risk, temporary search artifact, retrieved, verified documents, grounded
7.6 Variations and extensions
This section distinguishes HyDE’s published refinements from later variants and explains how multi-sampling, hybrid retrieval, and reverse indexing change its tradeoffs.
- Published HyDE vector: Explain that published HyDE averages embeddings from several hypothetical documents together with the original query embedding, rather than relying on one generated document.
Why it matters: This prevents you from mistaking multi-sampling or inclusion of the query for changes added only after the original HyDE paper. - Hybrid retrieval balance: Describe how Hybrid HyDE complements a HyDE embedding with the original query embedding or fuses HyDE results with lexical search such as BM25.
Why it matters: It explains why a semantic search signal need not replace the user's exact wording. - Reverse HyDE timing tradeoff: Trace how Reverse HyDE moves generation from query time to indexing time and state the resulting latency-versus-indexing tradeoff.
Why it matters: It identifies when generation work is paid for and what cost is exchanged for faster queries.
Key terms: standard HyDE vector, hypothetical documents, query's own embedding, Multi-Sample HyDE, Hybrid HyDE, original query's embedding, lexical search, BM25, Reverse HyDE, query time
7.6.1 Multi-sample HyDE
This section explains how averaging embeddings from multiple hypothetical documents makes HyDE retrieval less dependent on any one generation, at the cost of more generations per query.
- Multi-sample embedding consensus: Explain how multi-sample HyDE averages the embeddings of N hypothetical documents, optionally including the query’s own embedding, and why this reduces the influence of an unusual or poor generation.
Why it matters: It explains why several imagined documents can give retrieval a more reliable vector than just one. - Generation cost and sample count: Describe the latency and token cost of generating N hypothetical documents per query, and distinguish the suggested 3-to-5 working default from a number fixed by the paper.
Why it matters: The robustness benefit has a per-query cost, so the suggested sample count should not be mistaken for a prescribed setting.
Key terms: multi-sample HyDE, hypothetical document, embeddings, query vector, N generations per query, latency, tokens, working default
7.6.2 Hybrid HyDE (Fusion)
This section explains how hybrid HyDE preserves the value of the original query while using HyDE’s semantic reach, and illustrates the tradeoff with query-vector positions.
- Hybrid HyDE combination methods: Describe two ways to combine HyDE with the original query: average their vectors, or run HyDE vector search alongside lexical search and fuse the ranked results. Explain why retaining exact-match signals matters for terms such as part numbers, statute references, and drug names.
Why it matters: This explains how to improve semantic matching without losing queries whose exact wording is important. - Query vector comparison: Describe how the figure positions direct, single-sample HyDE, multi-sample averaged, and hybrid fusion representations relative to relevant documents. Explain the depicted noise reduction from averaging multiple HyDE samples and the hybrid balance between semantic reach and precision.
Why it matters: The comparison makes clear why a single generated sample may be noisy and why the original query still contributes useful precision.
Key terms: Hybrid HyDE, vector averaging, hybrid search, BM25, rank fusion, Reciprocal Rank Fusion, direct embedding, single HyDE sample, multi-sample averaged embedding, hybrid fusion embedding
7.6.3 HyDE with re-ranking
This section explains how HyDE can retrieve initial candidates for a more precise reranker, limiting the cost of that second stage.
- Two-stage HyDE retrieval: Describe how HyDE first retrieves a broad candidate set and a slower, more accurate model reranks only those candidates when high precision is needed.
Why it matters: You can improve match quality without applying the expensive model to every document. - Joint-input cross-encoder reranking: Explain how the cross-encoder reranker considers a query and candidate document together, in contrast to comparing vectors computed separately.
Why it matters: This distinction explains what kind of model can provide the more careful second-stage ranking.
Key terms: HyDE, multi-stage retrieval, candidate documents, re-ranking, cross-encoder, reranker
7.6.4 Reverse HyDE
This section explains how generating questions for documents at indexing time makes retrieval a question-to-question search and shifts LLM work off the query path.
- Question indexed document retrieval: Trace how Reverse HyDE indexes generated questions and uses a user's question to retrieve their source documents. Explain why matching questions to questions addresses the query-document mismatch.
Why it matters: It lets a conventional vector search match text in the same question-like form on both sides while still returning documents. - Indexing versus query cost: Explain how Reverse HyDE shifts generation work from query time to indexing time and when that tradeoff is attractive.
Why it matters: The benefit depends on whether avoiding generation during frequent searches outweighs extra indexing work and storage. - Lexical and embedding variants: Distinguish the section's BM25 document-expansion approach from its embedding-based question-row approach.
Why it matters: The distinction clarifies what must be indexed for generated questions to help in lexical versus vector retrieval.
Key terms: Reverse HyDE, hypothetical questions, query-to-query similarity search, metadata, deduplicating, query-time latency, indexing, hot path, generation, index
7.6.5 Learned query refinement: RQ-RAG
This section contrasts RQ-RAG’s learned query refinement with HyDE’s prompt-only approach, then explains practical ways to deploy HyDE and guard against misleading retrieval.
- Learned query refinement actions: Explain how RQ-RAG makes query refinement a trained model capability rather than a runtime prompt wrapper, and describe the four actions it can select.
Why it matters: The distinction is whether query handling comes from prompting an off-the-shelf model or training one to choose refinements. - Chained multi-hop refinements: Trace how RQ-RAG handles a multi-hop question by retrieving and answering sub-queries, feeding partial answers into later refinements, and selecting among candidate trajectories.
Why it matters: A multi-hop question may require an intermediate answer before the next useful search can be formulated. - RQ-RAG versus HyDE trade-off: Distinguish RQ-RAG’s reported gains and training cost from HyDE’s zero-shot, prompt-only convenience.
Why it matters: Better query control may justify training a custom model, but it is not a free replacement for HyDE. - Soft HyDE topical fingerprint: Explain why Soft HyDE can use an outline, bullets, or a short pseudo-passage instead of a polished hypothetical answer.
Why it matters: A shorter hypothetical can preserve the retrieval signal while reducing generation cost. - Practical HyDE pipeline additions: Describe how query rewriting, hybrid retrieval with BM25, re-ranking, and caching complement HyDE in deployment.
Why it matters: These additions address wording, exact-match, ranking, and repeat-query costs without changing HyDE’s basic idea. - Confidently wrong HyDE retrieval: Explain how a mistaken hypothetical can produce confidently wrong retrieval and what downstream checks the section recommends.
Why it matters: A strong similarity match does not establish that the imagined framing or retrieved evidence is correct.
Key terms: RQ-RAG, query refinement, control tokens, multi-hop, sub-query, refinement trajectories, trajectory selection, supervised fine-tuning, zero-shot, HyDE
7.7 Summary
Where this is used later
- The Reciprocal Rank Fusion algorithm (chapter 8) · Develops the Reciprocal Rank Fusion algorithm named here as a way to merge ranked results.
Concepts explained in this chapter
- hypothetical documentA hypothetical document is a synthetic passage generated to provide a document-like semantic representation of a user's information need.
- HyDEHyDE (Hypothetical Document Embeddings) is a retrieval technique that generates a hypothetical document or answer and embeds it to improve document retrieval.
- document-to-document similarityA retrieval comparison that evaluates similarity between a generated document-like passage and candidate documents rather than directly comparing the query with documents.
- zero-shot learningZero-shot retrieval is retrieval performed without relevance-labeled examples or task-specific fine-tuning of the retriever.
- semantic gapA semantic gap is a discrepancy between the meaning or representation of an input and the meaning or representation of a relevant target.
- ContrieverContriever is a dense retrieval model that learns representations for retrieving relevant documents without supervised retrieval labels.
- HydeRetrieverHydeRetriever is a LangChain retriever component that combines hypothetical-document embedding with vector-store retrieval behind a single query interface.
- HypotheticalDocumentEmbedderHypotheticalDocumentEmbedder is a LangChain component that generates hypothetical text and encodes it into an embedding for retrieval.
- bottleneck layerA dense bottleneck is a constrained continuous vector representation through which an encoder compresses input information.
- contrastive learningContrastive learning trains representations by bringing related examples closer and pushing unrelated examples farther apart in representation space.
- lossy compressionA lossy compressor produces a smaller representation by discarding some information from the original input.
- normalized discounted cumulative gainNDCG@10 is normalized discounted cumulative gain computed over the ten highest-ranked retrieval results.
- pseudo-supervisionPseudo-supervision is training or retrieval guidance supplied by synthetic signals generated from an existing model rather than direct human labels.
- vector databaseA vector database is a database system designed to store and retrieve vector representations using similarity search.
- similarity searchSimilarity search finds items whose representations are close to a query representation according to a similarity or distance function.
- document encoderA document encoder is a model component that transforms documents or passages into representations used for indexing and retrieval.
- RQ-RAGRQ-RAG is a retrieval technique that trains a language model to decide whether to retrieve, rewrite, decompose, or disambiguate a query.
- query refinementQuery refinement is the transformation of an information need into a more effective retrieval query or sequence of sub-queries.
- soft hydeSoft HyDE is a HyDE variant that generates a concise outline, keyword expansion, or bulleted pseudo-passage instead of a polished hypothetical document.
- multilingual supportA multilingual language model understands and generates natural-language text across multiple languages.
- hybrid searchHybrid search combines dense and sparse retrieval methods to improve the range and quality of retrieved results.
- reciprocal rank fusionRank fusion is a retrieval technique that combines ranked result lists from multiple searches into a single ranking.
- rerankingA reranker is a model or retrieval component that reorders candidate results according to estimated relevance.
- cross-encoderA cross-encoder jointly processes two input sequences through one encoder so that tokens from both sequences can interact directly during encoding.
- query rewritingQuery reformulation is the transformation of an existing search query into an alternative formulation intended to retrieve more suitable information.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
