Retrieval-augmented generation (RAG) gives an AI agent relevant information when it needs to answer or decide. Instead of placing an entire document collection in a prompt, the agent searches external storage and adds selected results to its current context. The store can grow independently of the model, but each model call is limited by its context window. section 1.3.4 section 6.1
RAG can retrieve both knowledge and memory. Knowledge may be documents the model was not trained on; memory may be previous conversations, user preferences, or agent experiences. Both use a retrieve-then-generate pattern, although their storage and retrieval needs can differ. section 1.3.4 section 6.1
A common document-chat workflow has two phases:
- Ingest: load documents, split them into useful chunks, turn the chunks into vectors with an embedding model, and store them in a searchable database. section 6.1.1
- Retrieve and answer: embed the user’s query, find similar chunks, add them to the prompt, and ask a separate language model to produce the response. section 6.1.1
Figure 6.2 RAG can be broken into two phases: ingestion and retrieval. First, documents are loaded, transformed, embedded, and stored in a searchable datastore. Second, they can be retrieved using query mechanisms as part of context augmentation.
The diagram separates preparation of searchable documents from retrieving chunks for a particular answer. section 6.1.1
A document question
For a query such as “capital of France,” an embedding model converts the query into a vector. A similarity search can return a stored chunk saying that Paris is the capital of France; the language model then uses that chunk as answer context. The embedding model finds relevant text, while the language model writes the answer. section 6.1.1
Table 6.1 Sources, storage formats, and retrieval mechanisms of knowledge and memory used in agent applications
| Source |
Storage format |
Retrieval mechanism |
| Memory (short-term)—previous conversation thread |
Agent/user conversation thread stored as text |
Provided automatically as message history |
| Memory (long-term)—previous user conversations |
Summarized memories stored as text or in a vector database (semantic) |
Semantic/vector search |
| Knowledge (unstructured), documents (PDFs) |
Vector database (semantic) and text document index |
Semantic/vector search and keyword search |
| Knowledge (structured), database tables, structured logs |
Same format as source |
Relational database queries, log parsing |
The table shows why retrieval need not mean vector search: conversation history, documents, database tables, and logs call for different storage and lookup methods. section 6.1
For a dependable agent, design retrieval around the information and the user’s access:
- Combine semantic search with keyword, relational, or graph queries where exact terms, numbers, fresh updates, or relationships matter. section 6.6
- Use coherent chunks and metadata filters; keep information current, and partition sensitive content by role or tenant. section 11.1.4
- Require answers to cite retrieved passages and allow “don’t know” when the answer is absent. A support agent can retrieve manuals and policies, then escalate uncertain or sensitive cases to a person. section 6.6 section 11.2
