liveBook · Manning

Embedding-based retrieval

As explained in Retrieval Augmented Generation, The Foundational Ideas

A bi-encoder is a neural architecture that independently encodes two inputs into representations that can be compared afterward.

A question and the passage that answers it may use different words. Embedding-based retrieval searches for passages by comparing vector representations rather than requiring an exact keyword match. That helps with questions such as “What causes rainfall?”, but a dense match can still miss a passage containing a crucial exact part number. section 4.1.1

A bi-encoder encodes the query and each document independently, then compares their embeddings, often with a dot product. Document vectors can be computed offline and stored in an index; a live query needs only its own embedding and a vector search. Unlike a cross-encoder, it does not run the full model on every query–document pair. That speed comes at the cost of comparing separate summaries rather than reading each pair together. section 2.3.1 section 8.6

Choose the encoder for the relationship being searched:

  • For paraphrases or duplicate questions, Sentence-BERT uses shared BERT weights and training that brings similar sentences together in vector space. section 2.3.1 section 3.1
  • For a short question seeking a longer answer passage, DPR uses specialized question and passage encoders trained on correct and incorrect passages. A model trained to find sentence twins may fail at this asymmetric task. section 3.1
  • When an embedding model specifies different query and document formats, follow them before indexing. For example, omitting BGE’s query prefix can silently worsen retrieval. section 3.1
Table 3.1 Symmetric search vs asymmetric search
  Feature  
  Symmetric Search  
  Asymmetric Search  
 Goal 
 Find similar text (twins) 
 Find answers to questions 
 Input 
 Similar length/structure 
 Short query vs. Long doc 
 Model 
 SBERT 
 DPR 
 Use Case 
 Paraphrase detection 
 Open Domain QA 

The symmetric-versus-asymmetric comparison helps distinguish sentence similarity from question-to-answer-passage retrieval. section 3.1

Scoring candidates

Given a batch of query embeddings and candidate-document embeddings, `tf.einsum("BD,BND->BN", query_emb, unflattened_candidate_emb)` produces one dot-product relevance score per candidate. The scores express vector alignment; they do not establish that a passage actually contains a usable answer. section 3.3 section 3.2

Retrieval-augmented systems can use these scores to select context for a reader or generator. REALM trains retrieval toward documents that help predict masked text, but needs a warm start because a random retriever supplies no useful learning signal. A modular RAG setup instead starts with pretrained DPR, keeps its document index fixed, and fine-tunes the query encoder. For higher precision, retrieve a small candidate set first and let a cross-encoder rerank it; combine keyword search when exact terms matter. section 3.2 section 4.1 section 8.6 section 4.1.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains independent encoding, offline indexing, similarity scores, training objectives, and reranking tradeoffs; walks REALM scoring code rather than training it from scratch.
Examples
Compares paraphrase search with open-domain question answering and computes candidate dot-product scores in a TensorFlow excerpt.
What it assumes
Builds from BERT’s token representations and explains why naive sentence pooling is not a reliable similarity embedding.
What you can do after
Evaluate/understand when to use shared or specialized encoders, precompute an index, and add keyword retrieval or reranking.
How it connects
Connects Sentence-BERT, dense passage retrieval, cross-encoders, approximate nearest-neighbor search, keyword search, and retrieval-augmented generation.

Also explained in

Compare how 4 books explain embedding-based retrieval →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.