A chatbot may recognize that a customer is asking about side effects yet miss the detail that matters: whether a particular painkiller is suitable alongside blood pressure medication. Adding a fixed answer for every variation is burdensome; returning search results leaves the customer to piece together an answer. Retrieval-augmented generation (RAG) offers another route: find relevant passages, then use a generative model to turn them into a direct answer grounded in those passages. section 6.1 section 6.2.1
From a generic reply to a grounded one
Suppose someone asks, “Can I take ibuprofen with my blood pressure medication? My arms are sore after getting the vaccine.” An intent-based bot might return general side-effect guidance, while search might display several passages. RAG retrieves pertinent passages and synthesizes a response addressing the medication concern. The answer can lead, with links to supporting documents available for verification. In a health-related conversation, that convenience is no reason to answer beyond what the sources support. section 6.1 section 6.2.1
A typical RAG flow has two phases—retrieval and generation:
- Prepare a searchable knowledge base. Convert documents into usable text, divide them into coherent chunks, and index them. One approach represents chunks as embeddings in a vector database. section 6.3.2
- Search for passages relevant to the user’s question and rank the results. A chatbot may first answer a familiar, confidently recognized intent without searching. section 6.3.1
- Give the question and retrieved results to a large language model. It generates a concise answer rooted in the documents, rather than merely displaying links and snippets. section 6.3.1
RAG is especially useful for uncommon, long-tail questions that would require many manually maintained intent–response pairs. Updating the documents can update the information available to the chatbot without adding another intent. It also differs from asking an LLM alone: the retrieved sources can contain current, domain-specific information unavailable in the model’s training data and make the basis of an answer traceable. RAG can complement predefined answers and task-oriented flows; an informational answer may also precede a transaction or human handover. section 6.0 section 6.4.1 section 6.2.3
Grounding reduces hallucination risk but does not remove it. Poor documents, irrelevant retrieval, or generation that strays from the passages can still produce a convincing wrong answer. If retrieval finds no suitable evidence, do not generate from weak results: say that a clear answer was not found and offer a human agent. Check performance at three levels—whether the document index is useful, retrieval finds the right passages, and the final answer addresses the question, fits the user’s context, and stays faithful to those passages. Human review can verify automated assessments. section 6.2.1 section 6.4.5 section 6.5
Table 6.2 Comparing the capabilities of intent-based chatbots, chatbots integrated with search, and RAG-integrated chatbots
|
Requirements
|
Intent-based chatbots
|
Chatbots integrated with search
|
RAG-integrated chatbots
|
|---|---|---|---|
|
Flexibility in handling queries
|
Limited to short-head predefined intents. May ignore nuance.
|
Handle long-tail queries by returning links and snippets
|
Handle long-tail queries by returning answers
|
|
Accuracy and relevance of responses
|
When an intent is recognized with high confidence, the answers are accurate and precrafted.
|
Provide contextually relevant and accurate documents that help the user find an answer
|
Provide contextually relevant and accurate answers grounded in your documents
|
|
Adding new know-ledge to the bot
|
Add or revise manually curated intent-response pairs
|
Add or revise documents in your knowledge base
|
Add or revise documents in your knowledge base
|
|
Maintenance and scalability
|
Extensive regression testing when intent training data is changed
|
Document repository needs to be maintained by adding new documents and removing stale documents
|
Document repository needs to be maintained by adding new documents and removing stale documents
|
|
Response generation quality
|
Predefined responses are presented.
|
User must put together their own answer from retrieved passages and documents
|
Answers are grounded in source documents but adapted to nuance from the question
|
Compare intent-based chatbots, search-integrated chatbots, and RAG-integrated chatbots by the answers they return and how new knowledge is added. section 6.2.4
