chapter two

2 Revolutions in semantics, scale, and similarity

 

This chapter covers

  • Word2Vec's breakthrough in semantic understanding
  • FAISS's solution to billion-scale similarity search
  • Sentence-BERT's practical transformer similarity
  • How these technologies have converged for RAG

It's important to remember that RAG is not a particular technology but an architectural pattern. By 2020, three separate technological innovations converged to pave the way for what would become retrieval-augmented generation, RAG.

The first of them was Word2Vec, a model that proved that semantic relationships could be captured mathematically, in vector form. Traditional search systems struggled when keywords didn't match (vocabulary mismatch or lexical mismatch), but this could address it, and later Transformer models showed how to generalize this idea to larger contexts. FAISS (Facebook AI Similarity Search), an approach and a library that made billion-scale similarity search practical, made high-dimensional vector search viable for production infrastructure. Finally, Sentence-BERT, which adapted Transformers for real-time similarity applications, completed the pipeline from user queries to relevant document retrieval.

2.1 Word2Vec (2013): Semantic word embeddings

2.1.1 From search engines to semantic understanding

2.1.2 Word2Vec: From words to vectors

2.1.3 Implementing Word2Vec: From theory to practice

2.1.4 Business impact

2.1.5 The foundation for dense retrieval

2.1.6 Limitations and the path forward

2.2 FAISS (2017): Billion-scale similarity search

2.2.1 Approximate nearest neighbors

2.2.2 The scale advantage

2.2.3 The bridge to practical semantic search

2.3 Sentence-BERT (2019): Practical sentence similarity

2.3.1 From cross-encoders to bi-encoders

2.3.2 Implementation and business impact

2.3.3 Training, performance, and limitations

2.3.4 The final piece of the RAG puzzle

2.4 Ready for revolution

2.4.1 The modern retrieval stack

2.4.2 From retrieval to synthesis

2.5 Summary