liveBook · Manning

Similarity search

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Information retrieval is the process of identifying and extracting relevant resources from large data collections in response to an information need.

Similarity search retrieves documents that are semantically relevant to a query, even when they use different wording. Dense vector search is useful for conceptual questions and paraphrases, but it can miss precise identifiers such as error codes or contract clause numbers. section 6.2.1 section 6.5

In retrieval-augmented generation (RAG), retrieved documents enter the model’s context so it can answer using external knowledge. Retrieval and generation can fail independently: a model may summarize irrelevant documents faithfully, or hallucinate despite receiving relevant ones. section 6.1 section 6.1.1

Choose and evaluate retrieval methods against the question being asked:

  • Use dense vector search for semantic similarity and paraphrases; check whether it misses exact terms or identifiers. section 6.2.1 section 6.5
  • Use BM25 for precise terminology and grep or ripgrep for exact strings or regular expressions. Exact matches are not necessarily relevant to the task. section 6.2.1
  • Consider hybrid retrieval to combine sparse and dense signals. Check for poor fusion weights, duplicates, and conflicting rankings. section 6.2.1 section 6.5
  • Measure precision and recall, then inspect ranking with metrics such as precision@k and nDCG. Relevant documents ranked below the context-window cutoff cannot influence generation. section 6.5 section 6.2.1

Diagnosing a retrieval gap

A financial assistant must answer what drove Q3 revenue growth and what risks could affect Q4. If the retrieved context covers the growth driver but not the risks, a faithful answer may still be incomplete. Check context recall and investigate whether the retrieval method missed relevant documents before changing generation. section 6.0 section 6.5

Table 6.2 Retrieval approaches and their failure modes
Retrieval method
Strength
Failure modes
Useful diagnostic signals
Grep / ripgrep
Exact, deterministic matching
Lexical variation; exact matches that are not task relevant
Recall across term variations; precision of exact matches
Sparse / TF-IDF / BM25
Fast, interpretable, and strong on   terminology
Paraphrases and synonyms
precision@k, recall@k, AP/MAP
Dense / vector search
Semantic similarity, handles paraphrases
Precise identifiers, exact terms
recall@k, nDCG with graded relevance, hybrid-vs-dense comparisons
Hybrid retrieval
Combines sparse and dense signals
Fusion weights, duplicate results, conflicting rankings
precision@k, recall@k, MAP, nDCG

Compare retrieval approaches, their failure modes, and useful diagnostic metrics. section 6.2.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Compares retrieval paradigms and their failure modes, with diagnostic metrics, without detailing a vector-search implementation.
Examples
Uses a financial research assistant answering questions about quarterly revenue and risks to illustrate incomplete retrieved evidence.
What you can do after
Diagnose retrieval gaps by comparing method-specific failure modes and inspecting precision, recall, and rank-aware metrics.
How it connects
Connects dense retrieval with semantic relevance, hybrid retrieval, retrieval-augmented generation, precision, recall, and ranking metrics.

Also explained in

Compare how 37 books explain similarity search →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.