A coding assistant may know Python yet know nothing about your organization’s changing codebase. Retrieval-augmented generation (RAG) supplies relevant information at request time instead of requiring a model to be fine-tuned whenever that information changes. It can also ground answers in documents, improving accuracy and reducing hallucinations—but retrieved context is not a guarantee of a better answer. section 10.2 section 6.1.7 section 10.4
A typical vector-database workflow has two parts:
- Prepare the data: split documents or code examples into chunks, embed them, and store the embeddings with their source content. section 6.1.7 section 10.2.2
- Handle a request: embed the input, run a similarity search, retrieve relevant content, and add content that fits the model’s context window to the prompt before generation. section 6.1.7 section 10.2.3
Figure 6.4 RAG system demonstrating how we use our input embeddings to run a search across our documentation, improving the results of the generated text from our LLM
The diagram connects input embeddings, a search across documents, and the context supplied to the generating model. section 6.1.7
Chunk size and result count are prompt-budget decisions. Splitting Wikipedia pages into 400-token chunks and retrieving three results adds up to 1,200 tokens before accounting for the user’s input and other instructions. A coding model with a 2,048-token context window has even less room for long code examples; candidate examples must be checked against the available space. section 6.1.7 section 10.1 section 10.2.3
From retrieval to a coding copilot
A Python coding dataset is embedded and indexed in Milvus. The API embeds an incoming request, searches for candidate examples, and inserts instruction-and-output pairs into a prompt for DeciCoder. This can expose the model to relevant coding patterns without fine-tuning it. Test the search results before relying on the generated code: relevant-looking matches do not establish that the complete prompt will work well. section 10.2.2 section 10.2.3 section 10.4
Where retrieval runs changes the application boundary. Backend RAG gives developers consistent control over context use and keeps the vector database less directly exposed, though prompt injection remains a concern. Frontend RAG can add business context to a generic LLM without fine-tuning. Either way, inspect what was retrieved and what was actually sent to the model. section 8.1.5 section 10.4
RAG can make results worse. In one coding-copilot test, the model without RAG correctly reversed a linked list, while the RAG-assisted result reversed an ordinary list. The retrieved examples looked relevant, but their plain-English instructions differed from users’ half-written code; DeciCoder was already trained to complete code from code-like input. Compare against a no-RAG baseline rather than assuming more context helps. section 10.4 section 10.1
RAG is a form of grounding: it retrieves context to support generation. Although retrieval could use other search methods or databases, the term commonly implies semantic search over a vector database. That approach suits simple questions better than questions requiring several linked lookups; a knowledge graph is one possible way to represent the relationships those questions need. section 12.2.5
