chapter three
3 REALM: Birth of end-to-end trainable RAG
This chapter covers
- The challenges of open-domain question answering
- From symmetric semantic search to asymmetric, task-specific retrieval
- REALM’s innovation: joint pre-training
- Training a retriever with a language model’s objective
In the previous chapter, we assembled the parts of a scalable retrieval system: semantic understanding from Word2Vec, sentence embeddings from Sentence-BERT, and large-scale search using libraries like FAISS. These systems can retrieve documents, but when it comes to generating a response, they leave the user with the burden of reading all the documents, digesting them, and extracting an answer. In this chapter, we focus on how the retriever can be optimized for utility or relevance (rather than query-document similarity), although we already see the extractive capabilities of these systems.