An agent can answer from the current conversation, but useful context may live elsewhere: in documents, databases, or earlier interactions. Knowledge and memory are the resources that let it bring that information into a task. Knowledge is usually external information, such as a PDF or database table. Memory captures experiences, such as past conversations, user preferences, and agent actions. section 6.1 section 6.6
Think of storage and use as separate steps. An external store can grow independently of the language model, but only a limited amount fits in one prompt. Retrieval searches the store for relevant pieces; augmentation places those pieces in the prompt. Together, these steps are retrieval-augmented generation (RAG). section 6.1
Table 6.1 Sources, storage formats, and retrieval mechanisms of knowledge and memory used in agent applications
| Source |
Storage format |
Retrieval mechanism |
| Memory (short-term)—previous conversation thread |
Agent/user conversation thread stored as text |
Provided automatically as message history |
| Memory (long-term)—previous user conversations |
Summarized memories stored as text or in a vector database (semantic) |
Semantic/vector search |
| Knowledge (unstructured), documents (PDFs) |
Vector database (semantic) and text document index |
Semantic/vector search and keyword search |
| Knowledge (structured), database tables, structured logs |
Same format as source |
Relational database queries, log parsing |
Compare each information source with its storage format and retrieval method. A conversation thread, a PDF collection, and a database table need not be searched in the same way. section 6.1
Choose a retrieval path that fits the information:
- Keep the current conversation as short-term memory, supplied as message history. Store useful information from older conversations as long-term memory that can be searched later. section 6.1
- Search documents with semantic or keyword search. Query structured tables with database queries rather than treating every source as a document. section 6.1
- Combine semantic and keyword search when both meaning and exact wording matter. Use graph queries for relationships that similarity search alone may not resolve. section 6.6
A question that needs older context
Suppose a user asks a question that depends on an earlier conversation and a PDF. The current thread can supply recent context; retrieval can find a relevant saved memory and passage from the PDF. The agent can then answer using those retrieved pieces and cite the passage, rather than relying on an unsupported recollection. section 6.1 section 6.6
Treat knowledge and memory as one capability layer alongside persona, tools and actions, reasoning and planning, and evaluation and feedback—not as a fixed step in a runtime sequence. For reliable use, keep session and long-term memory separate, prune stale or repetitive items, make retrieval calls observable, respect access boundaries, and let the agent say “don’t know” when retrieved evidence is insufficient. section 1.3 section 11.5
