liveBook · Manning

Information retrieval

As explained in Retrieval Augmented Generation, The Foundational Ideas

Retrieval finds candidate documents or passages from a knowledge base in response to a query.

A customer asking about warranty coverage needs the applicable policy, not ten PDFs to inspect or an answer based on possibly outdated model knowledge. Information retrieval finds candidate documents or passages in a knowledge base so a system can locate evidence relevant to the question. External retrieval also crosses a model’s knowledge boundary: a policy update or yesterday’s port closure cannot be recovered from frozen model weights alone. section 1.2 section 1.3.1

In a retrieval-augmented generation (RAG) system, indexing and querying are separate jobs. Offline indexing chunks documents, converts them into searchable representations, and stores them. At query time, query processing prepares the question for search; retrieval fetches a broad candidate set; ranking filters and orders those candidates; and generation uses the selected evidence to answer. A relevant passage supplied with an answer also lets a reviewer check its claims. section 1.2 section 1.3.5

Table 2.1 Timeline of information retrieval
Era/Phase
Year
Key Concept
Seminal Paper/Event
Boolean / Keyword Search
Pre-1970s
Earliest methods of structured text querying using exact match and set logic (AND/OR/NOT)
Based on logical methods
Inverted Index
1970s
Data structure that maps terms to documents, enabling fast, large-scale full-text search
 
Vector Space Model (VSM) and TF–IDF
1975
Concept of relevance ranking by representing queries and documents as weighted vectors
G. Salton, A. Wong, and C. S. Yang, "A Vector Space Model for Automatic Indexing"
Okapi BM25
1994
Probabilistic model that refined ranking by introducing document-length normalization and term saturation
S. E. Robertson et al., "Okapi at TREC-3"
Locality-Sensitive Hashing (LSH)
1998
Foundation for Approximate Nearest Neighbor (ANN) search to efficiently find similarities in high-dimensional vector spaces
P. Indyk and R. Motwani, "Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality"
Word2vec / Dense Word Embeddings
2013
Dense vector representations of words learned from context, capturing semantic relationships geometrically (e.g., King - Man + Woman ≈ Queen)
T. Mikolov et al., "Efficient Estimation of Word Representations in Vector Space"
FAISS (Facebook AI Similarity Search)
2017
High-performance billion-scale implementations of ANN algorithms (e.g., HNSW, PQ), making dense retrieval feasible in real-world systems
H. Jégou, M. Douze, J. Johnson (Initial release, February 2017)
Transformers / BERT, Sentence-BERT (SBERT) / Bi-encoders
2018, 2019
Adapted Transformers to produce fixed-size sentence embeddings suitable for fast nearest-neighbor search via ANN indices, solving the latency problem of cross-encoders
J. Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers..."
Retrieval-Augmented Generation (RAG)
2020
Integration of retrieval systems with large language models (LLMs), using retrieved documents as an external memory source to prevent hallucination and provide up-to-date context.
L. Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" 

The timeline shows how retrieval moved from exact-match Boolean queries and inverted indexes through TF-IDF and BM25 to vector representations and scalable similarity search. These approaches remain design options rather than a mandate to replace keywords with vectors. section 2.0

Choose a retrieval approach by the failure it must address:

  • If needed content never reaches the candidate set, inspect indexing and retrieval before changing generation. section 1.2
  • If a useful candidate is found but buried, inspect ranking; BM25 and semantic methods offer different ways to assess relevance. section 1.2 section 2.0
  • If exact terms matter alongside meaning, consider hybrid search rather than discarding sparse keyword retrieval. section 2.0

Retrieval can extend beyond text. Multimodal systems may maintain separate text and image indices, while example-selection strategies can favor similarity or diversity. Such choices should serve the information the final context needs; faster retrieval alone does not turn documents into a coherent answer. section 13.8.3 section 2.4.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains offline indexing and online candidate retrieval, then surveys search techniques without specifying a complete implementation here.
Examples
Uses a warranty question, changing organizational knowledge, and a historical retrieval timeline to distinguish search needs and approaches.
What it assumes
Starts with Boolean search and inverted indexes before moving toward semantic retrieval and RAG architecture.
What you can do after
Evaluate/understand whether missing content calls for indexing or retrieval changes and whether buried candidates call for ranking changes.
How it connects
Links retrieval to inverted indexes, TF-IDF, BM25, semantic search, approximate nearest-neighbor search, ranking, and RAG.

Also explained in

Compare how 17 books explain information retrieval →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.