A customer-support chatbot may need to answer questions about products and policies that a pretrained language model does not know. Retrieval-augmented generation (RAG) supplies relevant information from a knowledge base when the chatbot answers, rather than relying on the model’s pretrained knowledge alone. section 10.0 section 10.7
Why not put every support article in every prompt? A model’s context window limits how much it can receive, very large prompts can degrade performance, and sending the whole knowledge base on each call increases token costs. RAG selects the parts needed for the current question. section 10.5.1
The basic flow is:
- Read the user’s question. section 10.5.1
- Retrieve relevant context from the knowledge base. section 10.5.1
- Add that context to the question and ask the language model to generate an answer. section 10.5.1
Support articles as a knowledge base
Text files describe products and fountain-pen maintenance. An ingestion script loads the files, splits them into overlapping chunks, and stores the chunks with embeddings in a Pinecone index. Similarity search can then return chunks related to a customer’s question. API keys and index settings are kept in Streamlit secrets. section 10.5.2
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)
texts = splitter.split_documents(documents)
The splitter uses chunks of 1,000 characters with 200 characters of overlap before the documents are stored. section 10.5.2
In the chatbot workflow, a retrieval step searches using the conversation’s message contents. An augmentation step joins the returned document contents and inserts them into the system message. The prompt also tells the chatbot to refer the customer to a human representative if it cannot answer confidently from the supplied information. section 10.5.2
@staticmethod
def augment_node(state: AgentState):
docs = state["retrieved_docs"]
docs_content_list = [doc.page_content for doc in docs]
content = "\n".join(docs_content_list)
new_text = SYS_MSG_AUGMENTATION.replace("{docs_content}", content)
return {"sys_msg_text": BASE_SYS_MSG + "\n\n" + new_text}
This augmentation step turns retrieved document contents into text for the system message. section 10.5.2
