chapter two

2 Revolutions in semantics, scale, and similarity

 

This chapter covers

  • Word2Vec's breakthrough in semantic understanding
  • FAISS's solution to billion-scale similarity search
  • Sentence-BERT's practical transformer similarity
  • How these technologies have converged for RAG

By 2020, the stage was set for a revolution in how AI systems access and use knowledge. Three seemingly unrelated technological innovations had quietly converged to create the perfect conditions for what would become retrieval-augmented generation.

Word2Vec, a model that proved semantic relationships could be captured mathematically, addressed the vocabulary mismatch that limited traditional search systems. FAISS (Facebook AI Similarity Search), an approach and a library that made billion-scale similarity search practical, turned high-dimensional vector search from a research curiosity into production infrastructure. Sentence-BERT, which adapted transformers for real-time similarity applications, completed the pipeline from user queries to relevant document retrieval.

Each technology solved a critical limitation that had prevented semantic search from becoming practical for enterprise applications. Together, they established the foundational infrastructure that made retrieval-augmented generation possible. Understanding these foundations is essential because modern RAG systems inherit both the strengths and failure modes of the technologies they’re built on.

2.1 Word2Vec (2013): Semantic word embeddings

2.1.1 From search engines to semantic understanding

2.1.2 Word2Vec: From words to vectors

2.1.3 Implementing Word2Vec: From theory to practice

2.1.4 Business impact

2.1.5 The foundation for dense retrieval

2.1.6 Limitations and the path forward

2.2 FAISS (2017): Billion-scale similarity search

2.2.1 Approximate nearest neighbors

2.2.2 The scale advantage

2.2.3 The bridge to practical semantic search

2.3 Sentence-BERT (2019): Practical sentence similarity

2.3.1 From cross-encoders to bi-encoders

2.3.2 Implementation and business impact

2.3.3 Training, performance, and limitations

2.3.4 The final piece of the RAG puzzle

2.4 Ready for revolution

2.4.1 The modern retrieval stack

2.4.2 From retrieval to synthesis

2.5 Summary