Retrieval-augmented generation

As explained in How Large Language Models Work

Retrieval-augmented generation is a language-generation approach that retrieves relevant documents and supplies them to a language model when producing an answer.

Retrieval-augmented generation (RAG) helps a language model answer by searching relevant documents first, then adding those documents to the user’s prompt. The model generates from that combined context rather than relying only on its training or fine-tuning data. section 5.5.1

A RAG system usually follows this path:

• Turns the user’s input into a search query. section 5.5.1
• Searches a general or subject-specific document index. section 5.5.1
• Combines the retrieved documents with the original question. section 5.5.1
• Asks the language model to produce an answer from that context, potentially with citations to the source documents. section 5.5.1

RAG can make answers more accurate, factually grounded, and useful because the model receives relevant source material. It does not guarantee correctness: the language model still writes the final response, so its errors remain possible. section 5.5.1 section 5.6

A small DSPy implementation

A DSPy RAG module can configure GPT-3.5 as the language model, ColBERTv2 over a Wikipedia database as the retriever, fetch three passages, and pass the question and passages to an answer generator. The components can later be replaced separately. section 5.5.2

Figure 5.11 On the left, we show the normal use of an LLM of a user asking about how to write JSON. LLMs naturally have the chance of producing errant outputs, which we want to minimize. On the right, we show the RAG approach. By using a search engine, we can find documents that are relevant to a query and combine them into a new prompt, giving the LLM more information and context to produce a better answer.

Compare ordinary language-model prompting with RAG: the latter inserts search results into a new prompt before generation. section 5.5.1

How How Large Language Models Work covers it

How deep it goes

Explains the retrieval-to-prompt pipeline and shows a lightweight implementation, but does not develop a full tutorial.

Examples

Uses a subject-specific document index, Wikipedia abstracts, GPT-3.5, ColBERTv2, and a three-passage DSPy module.

What you can do after

Build it from scratch as a document-retrieval and answer-generation pipeline, or implement it with DSPy.

How it connects

Connects language models with search engines, document databases, vectorization, prompting, and citations.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Related concepts

• similarity searchuses · Similarity search is a retrieval technique that finds or ranks items by comparing their representations with a query according to a similarity or distance measure.
• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• databaseuses · A database is an organized collection of persistent data managed by software for storage, retrieval, querying, and modification.
• DSPyimplemented by · DSPy is a Python framework for declaratively programming, composing, evaluating, and optimizing language-model applications and pipelines.
• automation biascontrasts with · Automation bias is the tendency to over-rely on automated outputs or recommendations without sufficient independent evaluation or verification.
• GPT modeluses · A GPT model is a generative pretrained Transformer language model that produces text autoregressively by predicting successive tokens from preceding context.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.