liveBook · Manning

HydeRetriever

As explained in Retrieval Augmented Generation, The Foundational Ideas

HydeRetriever is a LangChain retriever component that combines hypothetical-document embedding with vector-store retrieval behind a single query interface.

HydeRetriever packages hypothetical-document generation, embedding, and vector-store search behind one retriever interface. A query triggers an LLM to write a passage, encodes that passage, and uses the resulting HyDE vector for retrieval. section 7.3

The abstraction changes the retrieval path in three ways:

  • It hides embedder and vector-store coordination from application logic. section 7.3
  • It supports a one-line replacement for an existing retriever in production code. section 7.3
  • It can improve zero-shot retrieval over BM25 and standard unsupervised dense retrieval, while approaching supervised performance. section 7.3

Query-time flow

For “What are the best practices for a startup's user growth?”, the embedder generates a hypothetical answer passage, embeds it, and returns one vector for vector-store search. section 7.3

Listing 7.3 HyDE in LangChain.
from providers import get_chat_model, get_embeddings
from langchain_core.prompts import PromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_classic.chains.hyde.base import (
    HypotheticalDocumentEmbedder)
llm = get_chat_model(temperature=0)
base_embeddings = get_embeddings()
prompt = PromptTemplate(
    input_variables=["question"],
    template=("Please write a passage to answer the question.\n"
              "Question: {question}\nPassage:")
)  #A
hyde_embedder = HypotheticalDocumentEmbedder(
    llm_chain=prompt | llm | StrOutputParser(),  #B
    base_embeddings=base_embeddings,
)
query = "What are the best practices for a startup's user growth?"
hypothetical_doc_embedding = hyde_embedder.embed_query(query)  #C
 

Inspect the LangChain construction of the prompt, LLM chain, base embeddings, and embed_query call. section 7.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the generation-to-embedding-to-search pipeline and implementation wrapper, but not its internal framework mechanics.
Examples
Uses a startup user-growth query, FAISS similarity search, and BEIR benchmark comparisons.
What you can do after
Apply HyDE through a LangChain retriever while preserving the surrounding application architecture.
How it connects
Links hypothetical-document embedding with LLMs, vector stores, dense retrieval, BM25, and supervised retrieval.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.