liveBook · Manning

Vector space

As explained in Retrieval Augmented Generation, The Foundational Ideas

A vector space is a mathematical space whose elements are vectors that support operations such as addition and scalar multiplication.

A vector space gives semantic search a geometric foundation: queries and documents become vectors, and retrieval depends on how well their relationships are encoded. Poor representations create relevance gaps that later systems may mistake for generation failures. section 2.0

In retrieval systems, vector spaces support several linked design choices:

  • Vector Space Models represent queries and documents as weighted vectors for relevance ranking. section 2.0
  • Modern embedding models place text in high-dimensional spaces, such as 1,536 dimensions for text-embedding-3-small. section 10.3
  • Vector databases and approximate-nearest-neighbor indexes trade speed, recall, and memory when searching these spaces. section 2.0

Diagnosing retrieval quality

Suppose a query retrieves irrelevant context. Inspect the representation and search layer separately: semantic relationships may be poorly encoded in the vector space, or the ANN index may be sacrificing recall for speed or memory efficiency. section 2.0

Table 2.1 Timeline of information retrieval
Era/Phase
Year
Key Concept
Seminal Paper/Event
Boolean / Keyword Search
Pre-1970s
Earliest methods of structured text querying using exact match and set logic (AND/OR/NOT)
Based on logical methods
Inverted Index
1970s
Data structure that maps terms to documents, enabling fast, large-scale full-text search
 
Vector Space Model (VSM) and TF–IDF
1975
Concept of relevance ranking by representing queries and documents as weighted vectors
G. Salton, A. Wong, and C. S. Yang, "A Vector Space Model for Automatic Indexing"
Okapi BM25
1994
Probabilistic model that refined ranking by introducing document-length normalization and term saturation
S. E. Robertson et al., "Okapi at TREC-3"
Locality-Sensitive Hashing (LSH)
1998
Foundation for Approximate Nearest Neighbor (ANN) search to efficiently find similarities in high-dimensional vector spaces
P. Indyk and R. Motwani, "Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality"
Word2vec / Dense Word Embeddings
2013
Dense vector representations of words learned from context, capturing semantic relationships geometrically (e.g., King - Man + Woman ≈ Queen)
T. Mikolov et al., "Efficient Estimation of Word Representations in Vector Space"
FAISS (Facebook AI Similarity Search)
2017
High-performance billion-scale implementations of ANN algorithms (e.g., HNSW, PQ), making dense retrieval feasible in real-world systems
H. Jégou, M. Douze, J. Johnson (Initial release, February 2017)
Transformers / BERT, Sentence-BERT (SBERT) / Bi-encoders
2018, 2019
Adapted Transformers to produce fixed-size sentence embeddings suitable for fast nearest-neighbor search via ANN indices, solving the latency problem of cross-encoders
J. Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers..."
Retrieval-Augmented Generation (RAG)
2020
Integration of retrieval systems with large language models (LLMs), using retrieved documents as an external memory source to prevent hallucination and provide up-to-date context.
L. Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" 

Use the timeline to connect vector spaces with the progression from TF–IDF ranking to dense representations and scalable approximate-nearest-neighbor search. section 2.0

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the retrieval role and engineering trade-offs but defers mathematical proofs and detailed clustering mechanics.
Examples
Uses TF–IDF vectors, modern embedding dimensions, ANN indexes, and clustering text embeddings as concrete examples.
What you can do after
Evaluate retrieval representations and index choices against relevance, speed, recall, and memory trade-offs.
How it connects
Connects embeddings, semantic search, vector databases, approximate nearest-neighbor search, and clustering.

Also explained in

Compare how 6 books explain vector space →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.