liveBook · Manning

Retrieval-augmented generation

As explained in Machines that Think

Retrieval-augmented generation is a technique that retrieves external information and supplies it to a generative model as input context.

Retrieval-augmented generation (RAG) supplies selected external information to a generative model at the time of a query. Rather than relying solely on the model’s learned parameters, it places relevant source material in the prompt, where it can inform the answer without changing those parameters. section 6.2.1 section 6.4.4

A large knowledge source may not fit in a model’s context window. RAG addresses this limit by selecting a small set of passages for each question instead of placing the entire source in the prompt. New documents can be indexed and retrieved without retraining the model. section 6.4.4 section 6.5

One retrieval pipeline works as follows:

  • Convert the question into an embedding and compare it with document embeddings in an external knowledge base. section 6.4.4
  • Rank the documents by similarity and insert the original text of high-ranking passages into the prompt alongside the question. section 6.4.4
  • Generate an answer conditioned on the question, the retrieved passages, and the model’s learned parameters. section 6.4.4
Figure 6.8 Retrieval-augmented generation pipeline. A user query is transformed into an embedding, a vector representation used for similarity comparison against embedded documents in a knowledge base. The system ranks documents by semantic similarity and inserts only the top matches as text into the prompt before generation. The model’s response is thus conditioned on both its internal parameters and selectively retrieved external knowledge.

The pipeline distinguishes embeddings used for retrieval from the original text supplied to the model. section 6.4.4

Retrieval can reduce unsupported claims by bringing relevant evidence into view, but it does not guarantee a grounded answer. The retrieved material may be irrelevant or of poor quality; even relevant evidence can be underused, selectively interpreted, or extended with unsupported claims. Retrieved text remains subject to the model’s attention and positional limitations. section 6.2.1 section 6.4.5

Organizational knowledge

A corporate repository can be too large to fit in a single context window, making selective retrieval useful. In high-stakes deployments, retrieval alone is insufficient: data governance, traceability, access boundaries, integration, and regulatory compliance also matter. section 6.4.4 section 6.4.5

How Machines that Think covers it

How deep it goes
Explains an embedding-based retrieval pipeline and its limitations, while noting that RAG encompasses multiple system designs.
Examples
Uses legal archives, medical databases, and corporate repositories to illustrate knowledge sources too large for a single context window.
What you can do after
Evaluate retrieval as a reliability measure while recognizing that relevant evidence does not guarantee grounded generation.
How it connects
Connects retrieval to embeddings, context windows, attention, hallucination, and grounding.

Also explained in

Compare how 46 books explain retrieval-augmented generation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.