Suppose a chatbot must answer questions about a company’s documents. Putting every document into each prompt is impractical, and a general-purpose model cannot be assumed to know local information. Retrieval-augmented generation (RAG) solves this by finding relevant passages first, then giving those passages and the question to an LLM to generate an answer. The retrieved content provides grounding, but it does not guarantee a correct response. section 6.1.2 section 1.1.1
A basic RAG pipeline has two stages:
- Ingest and index: extract document text, split it into small chunks, create an embedding for each chunk, and store both chunks and embeddings in a vector store. Smaller chunks help searches find focused passages rather than large sections with mixed relevance. section 6.1.3
- Retrieve and generate: embed the user’s question, search for similar chunks, place the retrieved text alongside the question in a prompt, and send that prompt to the LLM. The LLM synthesizes the answer; the vector store supplies the context, not the prose. section 1.1.1 section 6.1.3
Figure 1.2 A Q&A engine implemented with RAG design. An LLM query engine stores domain-specific document information in a vector store. When an external system sends a query, it converts the natural language question into its embeddings (or vector) representation, retrieves the related documents from the vector store, and then gives the LLM the information it needs to craft a natural language response.
Follow the question through embedding, vector-store retrieval, and LLM answer generation; the documents are prepared before the question arrives. section 1.1.1
Questions about Paestum’s temples
A chatbot can retrieve stored passages about Paestum and pass them with a question to GPT-5-nano, rather than requiring the user to paste the full source text. One implementation reduces the query path to three function calls: query the vector database, assemble a prompt from the question and results, then execute the LLM prompt. Asked how many temples there are, who built them, and their architectural style, the demonstrated response identifies three Doric temples but says it does not know who constructed them. That is a useful boundary: check what the retrieved evidence supports instead of treating a fluent answer as proof. section 6.3 section 6.3.3
RAG keeps prompts shorter than passing entire documents and can reduce reliance on outdated or missing model knowledge. Its accuracy still depends on the source content, chunk boundaries, retrieval relevance, and model behavior. Poor splits can strand needed context; retrieving too few chunks can miss evidence, while retrieving more uses context space. A practical starting point is four results, followed by answer-quality testing. Use matching embedding dimensions during ingestion and querying; changing embedding models can require re-embedding the stored documents. section 1.4.2 section 6.4
To build the pipeline directly, connect an embedding model, a vector store, and an LLM for answer generation. ChromaDB and OpenAI provide one working combination; LangChain offers interchangeable loaders, splitters, embedding models, vector stores, and retrievers when the same pattern needs reusable components. section 6.4 section 1.7
