liveBook · Manning

Document-to-document similarity

As explained in Retrieval Augmented Generation, The Foundational Ideas

A retrieval comparison that evaluates similarity between a generated document-like passage and candidate documents rather than directly comparing the query with documents.

Document-to-document similarity reframes retrieval as a comparison between a generated document-like passage and corpus documents. HyDE uses an LLM to turn a short query into a hypothetical document whose embedding can match the corpus more effectively than the original query. section 7.0 section 7.2

Figure 7.1 With query–document mismatch, the query embedding lies far from the cluster of relevant documents, which is why direct embedding retrieval underperforms.

The query–document mismatch diagram shows why direct embedding retrieval can miss relevant documents when user and corpus vocabularies differ. section 7.1

A typical document-to-document retrieval pipeline separates relevance modeling from semantic search:

  • Generate a semantically plausible hypothetical document from the user's sparse query. section 7.2
  • Encode the generated passage and corpus documents with a document encoder. section 7.1
  • Rank corpus documents by embedding similarity, without explicitly scoring the original query against each document. section 7.1
  • Discard the generated passage after retrieval and ground the final answer in verified corpus documents. section 7.0

Medical vocabulary mismatch

For the query “why does my chest hurt when I climb the stairs?”, BM25 retrieves documents with little lexical overlap, while direct dense retrieval finds only one relevant document in its top two. A hypothetical passage using terms such as “substernal discomfort” and “exertion” retrieves both relevant documents, achieving Recall@2 of 1.00. section 7.1

Figure 7.2 HyDE Workflow: HyDE separates relevance modeling (LLM generation g(q)) from semantic encoding (f()) and retrieval over a corpus.

The workflow diagram clarifies the separation between LLM-based relevance modeling, document encoding, and corpus retrieval. section 7.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the two-step factorization, generation pipeline, implementation pattern, and Recall@2 comparison, but not retriever fine-tuning.
Examples
Demonstrates medical chest-pain retrieval with BM25, direct dense retrieval, HyDE, Python listings, and Recall@2 results.
What it assumes
Assumes familiarity with queries, documents, embeddings, and retrieval while explaining the vocabulary-mismatch problem from concrete examples.
What you can do after
Apply it via HyDE by generating, encoding, and searching a hypothetical document before factual answer synthesis.
How it connects
Connects HyDE with dense retrieval, BM25, query-document similarity, document encoders, and large language models.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.