A customer-service chatbot may need to answer a question about a return policy that is stored in company documents, not in the model’s training data. Retrieval-augmented generation (RAG) lets a language model look up relevant information before composing its answer. Instead of relying only on what it learned during training, the model can draw on documents or search results supplied as context. section 8.13 section 5.7
For a collection of documents, a typical RAG setup works as follows:
- Represent documents as embeddings—numerical representations intended to capture similarities in meaning—and store them in a vector database. section 8.13
- Compare a question with those representations to find relevant documents, then place passages from them in the model’s context before it answers. section 8.13
- Where appropriate, include sources with the answer so a person can check whether the retrieved information supports it. section 5.7 section 10.20
A question about travel expenses
An employee asks whether a premium economy ticket is covered for a five-hour sales trip. A RAG system can retrieve relevant travel-policy passages and use them to answer. The result still depends on finding the right documents and interpreting them correctly. section 8.13 section 5.7
Retrieval can also help with questions about events after a model’s training cutoff. It may reduce invented answers, but it does not guarantee accuracy: search results must be reliable, and a cited summary still needs checking. Retrieval is increasingly combined with longer context windows, memory, and other ways of handling information. section 5.7 section 10.20
Figure 8.9 The architecture of retrieval-augmented generation (RAG)
The architecture diagram provides a visual guide to the retrieval step and the model’s use of retrieved information when generating a response. section 8.13
