Retrieval-augmented generation (RAG) combines external-data retrieval with language-model generation. It supplies a pretrained model with relevant information from private or public sources so responses can be more accurate, current, and traceable. section 2.2 section 2.6
A RAG pipeline separates two operating phases:
- Offline preparation chunks documents, converts chunks into embeddings, and stores them with search indexes in a vector database. This phase repeats when the knowledge base changes. section 13.2 section 2.1.3
- Online retrieval embeds the user query, searches for semantically similar chunks, and adds the selected context to the prompt. section 13.2 section 13.5
- Generation passes the enriched prompt to a language model, which produces an answer; the system can compare that answer with retrieved sources for transparency. section 2.2 section 13.5
RAG is often preferable when information changes or was absent from initial model training. It usually avoids model training and extensive labeling, while source traceability can reduce hallucinations and improve interpretability. However, retrieval cannot compensate for a model’s inability to perform the underlying task, and large deployments incur vector-database and pipeline costs. section 2.5
Compared with fine-tuning, RAG specializes access to knowledge rather than changing model parameters. Fine-tuning can be stronger for complex patterns and specialized domain behavior, but generally requires more curated data, training, and machine-learning expertise. The choice depends on the use case. section 2.5
A small end-to-end implementation
Extract text from a PDF with PyMuPDF, detect sentence boundaries with spaCy, and group sentences into chunks. Encode chunks with Sentence Transformers, save their vectors, and retrieve the five highest-scoring chunks for a query such as “blocked KV-cache.” Format those chunks with the query, then pass the prompt to a local Phi-3 model through llama.cpp. section 13.2 section 13.5
Figure 13.3 The generic workflow of a RAG system
Use the workflow diagram to distinguish document preparation, vector storage, query retrieval, context enrichment, and response generation. section 13.2
Baseline RAG can struggle with multihop questions because semantic similarity may miss relationships or details distributed across documents. GraphRAG augments retrieval with a knowledge graph of entities and relationships, then uses community summaries to support broader retrieval and reasoning. section 14.1
Evaluate retrieval and generation separately: context precision measures retrieved relevance, context recall measures coverage, faithfulness checks factual support, and answer relevancy checks alignment with the query. section 14.1.2
