A customer asking about warranty coverage needs the applicable policy, not ten PDFs to inspect or an answer based on possibly outdated model knowledge. Information retrieval finds candidate documents or passages in a knowledge base so a system can locate evidence relevant to the question. External retrieval also crosses a model’s knowledge boundary: a policy update or yesterday’s port closure cannot be recovered from frozen model weights alone. section 1.2 section 1.3.1
In a retrieval-augmented generation (RAG) system, indexing and querying are separate jobs. Offline indexing chunks documents, converts them into searchable representations, and stores them. At query time, query processing prepares the question for search; retrieval fetches a broad candidate set; ranking filters and orders those candidates; and generation uses the selected evidence to answer. A relevant passage supplied with an answer also lets a reviewer check its claims. section 1.2 section 1.3.5
Table 2.1 Timeline of information retrieval
|
Era/Phase
|
Year
|
Key Concept
|
Seminal Paper/Event
|
|
Boolean / Keyword Search
|
Pre-1970s
|
Earliest methods of structured text querying using exact
match and set logic (AND/OR/NOT)
|
Based on logical methods
|
|
Inverted Index
|
1970s
|
Data structure that maps terms to documents, enabling
fast, large-scale full-text search
|
|
|
Vector Space Model (VSM) and TF–IDF
|
1975
|
Concept of relevance ranking by representing queries and
documents as weighted vectors
|
G. Salton, A. Wong, and C. S. Yang, "A Vector Space
Model for Automatic Indexing"
|
|
Okapi BM25
|
1994
|
Probabilistic model that refined ranking by introducing
document-length normalization and term saturation
|
S. E. Robertson et al., "Okapi at TREC-3"
|
|
Locality-Sensitive Hashing (LSH)
|
1998
|
Foundation for Approximate Nearest Neighbor (ANN) search
to efficiently find similarities in high-dimensional vector spaces
|
P. Indyk and R. Motwani, "Approximate Nearest
Neighbors: Towards Removing the Curse of Dimensionality"
|
|
Word2vec / Dense Word Embeddings
|
2013
|
Dense
vector representations of words learned from context, capturing semantic
relationships geometrically (e.g., King - Man + Woman ≈ Queen)
|
T. Mikolov et al., "Efficient Estimation of Word
Representations in Vector Space"
|
|
FAISS (Facebook AI Similarity Search)
|
2017
|
High-performance billion-scale implementations of ANN
algorithms (e.g., HNSW, PQ), making dense retrieval feasible in real-world
systems
|
H. Jégou, M. Douze, J. Johnson (Initial release, February
2017)
|
|
Transformers / BERT, Sentence-BERT (SBERT) / Bi-encoders
|
2018, 2019
|
Adapted Transformers to produce fixed-size sentence
embeddings suitable for fast nearest-neighbor search via ANN indices, solving
the latency problem of cross-encoders
|
J. Devlin et al., "BERT: Pre-training of Deep
Bidirectional Transformers..."
|
|
Retrieval-Augmented Generation (RAG)
|
2020
|
Integration of retrieval systems with large language
models (LLMs), using retrieved documents as an external memory source to
prevent hallucination and provide up-to-date context.
|
L. Lewis et al., "Retrieval-Augmented Generation for
Knowledge-Intensive NLP Tasks"
|
The timeline shows how retrieval moved from exact-match Boolean queries and inverted indexes through TF-IDF and BM25 to vector representations and scalable similarity search. These approaches remain design options rather than a mandate to replace keywords with vectors. section 2.0
Choose a retrieval approach by the failure it must address:
- If needed content never reaches the candidate set, inspect indexing and retrieval before changing generation. section 1.2
- If a useful candidate is found but buried, inspect ranking; BM25 and semantic methods offer different ways to assess relevance. section 1.2 section 2.0
- If exact terms matter alongside meaning, consider hybrid search rather than discarding sparse keyword retrieval. section 2.0
Retrieval can extend beyond text. Multimodal systems may maintain separate text and image indices, while example-selection strategies can favor similarity or diversity. Such choices should serve the information the final context needs; faster retrieval alone does not turn documents into a coherent answer. section 13.8.3 section 2.4.2
















