A chatbot is a conversational interface that maintains context across exchanges. Unlike a task-focused engine that produces an answer and stops, it supports dialogue and follow-up questions. Unlike an agent, it does not need to plan and independently carry out multi-step actions. section 1.1
Start with a document and a question. Put both in a prompt, then ask an LLM to synthesize an answer from that context. For example, a question about Paestum’s temples can be answered from supplied text. This demonstrates document-grounded Q&A, but a single answer does not yet demonstrate conversational memory. section 6.1.1
Figure 6.1 The simple Q&A chatbot process involves the following: (1) the user sends a prompt containing a document (context) and a question to the chatbot, (2) the chatbot returns an initial answer, (3) the user follows up with a more detailed question, and (4) the chatbot provides a more accurate answer.
The diagram follows an initial document-grounded answer with a more detailed follow-up question—the interaction a conversational interface needs to support. section 6.1.1
To answer questions over a larger collection of documents, use retrieval-augmented generation (RAG):
- Prepare the content: extract text, split it into chunks, create embeddings, and store the chunks and embeddings in a vector database. section 6.1.3
- Answer a question: retrieve relevant content, combine it with the question in a prompt, and send that prompt to an LLM. section 6.1.3 section 6.3.3
A working retrieval chain can still be stateless. When asked a vague follow-up, it may not know what a word such as “they” refers to because previous questions and answers were not retained. Add chat history to the prompt and update it after each interaction. The resulting chain can use earlier turns to interpret the next question; retrieval supplies document context, while history supplies conversation context. section 7.3.4 section 7.4.4
From Q&A to a sustained conversation
A user first asks where Poseidonia was and who renamed it. A later question—“And then what did they do?”—fails with the memoryless chain. With exchanged messages retained, the chatbot interprets “they” as the Romans and produces a coherent follow-up answer. Test such ambiguous questions rather than assuming that a RAG chain remembers them. section 7.3.4 section 7.4.4
