Chapter 10: Creating a coding copilot project: Integrating an LLM service into VS Code with RAG
The chapter introduces a project to build a Python-focused coding copilot in VS Code using a pretrained coding model, an LLM service, and local retrieval-augmented generation.
After this chapter you will be able to
- Coding copilot project roadmap. Describe the planned path from deploying a coding model as an API, through setting up a local VectorDB for RAG, to building a VS Code extension that uses the LLM service.
- Pretrained model and Python scope. Explain why the project starts with an open source model already trained on coding problems, customizes it with RAG rather than finetuning, and focuses on Python instead of every language.
- RAG compatibility limits. Describe the two uncertainties in adding RAG to DeciCoder: its 2,048-token context window limits how many large code examples fit, and it was trained to complete functions from names and docstrings rather than instruction-tuned to use retrieved context.
- Local API generation flow. Trace the chosen deployment path: run DeciCoder and its tokenizer in a local FastAPI service, call that service from the VS Code extension, and use stopping criteria so generation stops after one function. Describe the simple prompt-and-response check used to verify the server.
- Organization-specific coding context. Explain that RAG supplies context from an organization’s changing codebase without finetuning the model, so a generally capable coding model can produce results better suited to that code, such as using its variable names and in-house dependencies.
- Local Milvus startup. Describe setting up Milvus standalone locally with Docker by downloading its startup shell script and then running that script. Explain that the script is a convenience because the underlying Docker run command is long.
- Milvus collection ingestion pipeline. Describe how the pipeline prepares and uploads the examples to Milvus, including the code-focused embedding model and collection schema.
- Similarity index configuration. Explain how the chosen Milvus index and distance metric support similarity search over the stored embeddings.
- Retrieval query sanity check. Trace how the section tests the populated collection with a query and judges whether its search results are useful.
- Retrieved few-shot examples. Trace how the generate endpoint encodes the user's prompt, searches the Milvus collection for nearest neighbors, and formats the results as few-shot examples alongside an instruction and the user's input.
- Context-window example filtering. Explain why and how the endpoint uses a token counter to filter retrieved examples before combining the examples, instruction, and user input into the final DeciCoder prompt.
- Manual activation and testing. Explain how the manifest activates the extension for the inline-suggestion trigger and binds Alt+S to it, rather than relying on automatic requests while typing. Describe how F5 launches a Development Host for testing.
- Inline suggestion provider. Describe replacing the template command with a registered inline suggestion provider that returns a completion item for display as ghost text.
- Selection and trigger handling. Trace how the provider checks for a manual trigger and highlighted code, moves a backward-to-forward selection’s cursor to the end by reselecting and retriggering, then obtains the selected text.
- Service request and suggestion loop. Trace how the provider sends highlighted code in an asynchronous fetch request to the existing LLM API and returns its response as an inline suggestion; describe the final hands-on test.
- RAG and no-RAG results. Describe how the extension’s linked-list result changed when RAG was disabled, and explain why the retrieved examples alone do not account for the poorer RAG result.
- Prompt and training mismatch. Explain why relevant plain-English examples can confuse DeciCoder when a user supplies half-written code in the editor. Relate that mismatch to DeciCoder’s code-completion training and lack of instruction tuning.
- Output improvement options. Describe the proposed experiments for improving the copilot: replace plain-English RAG instructions with example code, finetune DeciCoder for instruction data, or choose another model.
What it builds on
- Implementing Meta’s Llama (chapter 9) · Provides the contrast with the previous project’s focus on building a model.
- Evaluating code generators (chapter 4) · evaluating code generators
- Retrieval-augmented generation (chapter 6) · the purpose of retrieval-augmented generation
- Creating an LLM service with RAG and more (chapter 6) · serving an LLM through an API
- Vector databases (chapter 3) · The role of vector databases in LLM infrastructure.
Section by section
Introduction
The chapter introduces a project to build a Python-focused coding copilot in VS Code using a pretrained coding model, an LLM service, and local retrieval-augmented generation.
- Coding copilot project roadmap: Describe the planned path from deploying a coding model as an API, through setting up a local VectorDB for RAG, to building a VS Code extension that uses the LLM service.
Why it matters: This roadmap connects the model, retrieval system, and editor extension into one project. - Pretrained model and Python scope: Explain why the project starts with an open source model already trained on coding problems, customizes it with RAG rather than finetuning, and focuses on Python instead of every language.
Why it matters: These choices keep the project focused and make its customization easier to update.
Key terms: coding model, API, VectorDB, retrieval-augmented generation, VS Code extension, LLM service, open source model, coding problems, RAG, finetuning
10.1 Our model
The section chooses DeciCoder for a Python coding copilot, examines its limitations for RAG, and serves it through a local API that the VS Code extension can call.
- DeciCoder model choice: Explain why the project uses DeciCoder for Python code generation: it is a small, commercially usable open-source model trained on a language-filtered code dataset.
Why it matters: The model choice shapes both the copilot’s capabilities and its deployment options. - RAG compatibility limits: Describe the two uncertainties in adding RAG to DeciCoder: its 2,048-token context window limits how many large code examples fit, and it was trained to complete functions from names and docstrings rather than instruction-tuned to use retrieved context.
Why it matters: Retrieved examples help only if the model can fit and make use of them. - Local API generation flow: Trace the chosen deployment path: run DeciCoder and its tokenizer in a local FastAPI service, call that service from the VS Code extension, and use stopping criteria so generation stops after one function. Describe the simple prompt-and-response check used to verify the server.
Why it matters: The API provides a Python-based model service the extension can use before RAG is added.
Key terms: DeciCoder, Stack dataset, 1B parameters, RAG, context window, 2,048 tokens, instruction tuned, HumanEval, FastAPI, DeciCoder model and tokenizer
10.2 Data is king
The section explains why RAG can adapt a coding model to an organization’s code and previews the data and API work needed to do it.
- Organization-specific coding context: Explain that RAG supplies context from an organization’s changing codebase without finetuning the model, so a generally capable coding model can produce results better suited to that code, such as using its variable names and in-house dependencies.
Why it matters: A coding copilot needs to work with your code, not just code in general. - RAG data integration plan: Describe the planned sequence: set up a VectorDB, upload a Python coding dataset, and update the existing API to use them.
Why it matters: These are the pieces the section will connect to make the dataset available to the copilot.
Key terms: RAG, context, finetune, code base, VectorDB, Python coding dataset, API
10.2.1 Our VectorDB
This section chooses a local Milvus vector database for the coding copilot and explains how to start it before loading the dataset.
- Vector search storage choice: Explain why the project uses Milvus while acknowledging that a small dataset could instead be searched in memory with Faiss or USearch. Describe Milvus as an open source vector database that can run locally or scale to a cloud cluster.
Why it matters: The storage choice explains why the project sets up a database before working with its dataset. - Local Milvus startup: Describe setting up Milvus standalone locally with Docker by downloading its startup shell script and then running that script. Explain that the script is a convenience because the underlying Docker run command is long.
Why it matters: The database must be running locally before the project can load data into it. - Milvus stop and delete commands: Distinguish the script's Stop command, which stops the Milvus Docker container, from its delete command, which removes the stored data from the computer. State that neither needs to be run during initial setup.
Why it matters: Stopping the service and deleting its data have different consequences when cleaning up the project.
Key terms: Milvus, VectorDB, Faiss, USearch, similarity search, Milvus standalone, Docker, startup script, Stop command, delete command
10.2.2 Our dataset
The section builds and tests a Milvus collection of Python coding examples so the copilot can retrieve relevant context for its prompts.
- Alpaca Python dataset choice: Explain why the project uses a Python-code subset of Alpaca rather than an organization's GitHub code, and describe what each example provides.
Why it matters: The dataset supplies the coding examples that the copilot will retrieve. - Milvus collection ingestion pipeline: Describe how the pipeline prepares and uploads the examples to Milvus, including the code-focused embedding model and collection schema.
Why it matters: Retrieval requires the examples to be stored as searchable embeddings with their associated data. - Similarity index configuration: Explain how the chosen Milvus index and distance metric support similarity search over the stored embeddings.
Why it matters: The index determines how the copilot can find relevant examples efficiently. - Retrieval query sanity check: Trace how the section tests the populated collection with a query and judges whether its search results are useful.
Why it matters: A test query checks that ingestion and indexing produced relevant candidates before those results enter prompts.
Key terms: Alpaca dataset, Python coding challenges, synthetic data, PythonCodeIngestion, Milvus, collection, schema, metadata, CodeSearchNet, IVF_FLAT
10.2.3 Using RAG
This section adds retrieval-augmented generation to the coding copilot API by retrieving code examples, fitting them into the prompt, and testing the updated service.
- RAG service setup: Describe the API setup needed before retrieval: add imports, connect to Milvus, load the embedding model, and use a FastAPI lifecycle to load and release the collection.
Why it matters: The service needs the embedding model and collection available before it can retrieve examples. - Retrieved few-shot examples: Trace how the generate endpoint encodes the user's prompt, searches the Milvus collection for nearest neighbors, and formats the results as few-shot examples alongside an instruction and the user's input.
Why it matters: This is how retrieved project data becomes context for the coding model. - Context-window example filtering: Explain why and how the endpoint uses a token counter to filter retrieved examples before combining the examples, instruction, and user input into the final DeciCoder prompt.
Why it matters: Retrieved examples are useful only if the assembled prompt fits the model's context window.
Key terms: Milvus service, embedding model, FastAPI lifecycle, collection, generate endpoint, nearest neighbors, few-shot prompt example dataset, instruction prompt, limit, token counter
10.3 Build the VS Code extension
This section builds a VS Code coding copilot extension that sends selected code to an existing LLM service and displays the response as an inline suggestion.
- Extension scaffold and files: Describe how the VS Code extension generator creates a JavaScript project and distinguish the roles of package.json and extension.js.
Why it matters: The scaffold provides the files needed to configure the extension and implement its behavior. - Manual activation and testing: Explain how the manifest activates the extension for the inline-suggestion trigger and binds Alt+S to it, rather than relying on automatic requests while typing. Describe how F5 launches a Development Host for testing.
Why it matters: Manual triggering gives the user control and avoids unnecessary LLM-service requests. - Inline suggestion provider: Describe replacing the template command with a registered inline suggestion provider that returns a completion item for display as ghost text.
Why it matters: The provider supplies the VS Code interface for previewing and accepting generated code. - Selection and trigger handling: Trace how the provider checks for a manual trigger and highlighted code, moves a backward-to-forward selection’s cursor to the end by reselecting and retriggering, then obtains the selected text.
Why it matters: These checks prevent unwanted requests and put the suggestion where the user can see it. - Service request and suggestion loop: Trace how the provider sends highlighted code in an asynchronous fetch request to the existing LLM API and returns its response as an inline suggestion; describe the final hands-on test.
Why it matters: This connects the editor interaction to the chapter’s existing code model and RAG-backed service.
Key terms: Node, npm, VS Code extension generator, package.json, extension.js, activationEvents, onCommand:editor.action.inlineSuggest.trigger, contributes, keybindings, Alt+S
10.4 Lessons learned and next steps
The section diagnoses why RAG worsened the coding copilot’s output and proposes ways to improve or simplify the project.
- RAG and no-RAG results: Describe how the extension’s linked-list result changed when RAG was disabled, and explain why the retrieved examples alone do not account for the poorer RAG result.
Why it matters: Comparing the outputs helps pinpoint what to investigate rather than assuming retrieval is the problem. - Prompt and training mismatch: Explain why relevant plain-English examples can confuse DeciCoder when a user supplies half-written code in the editor. Relate that mismatch to DeciCoder’s code-completion training and lack of instruction tuning.
Why it matters: A useful prompt must fit both the model’s training and the way people actually use the editor. - Output improvement options: Describe the proposed experiments for improving the copilot: replace plain-English RAG instructions with example code, finetune DeciCoder for instruction data, or choose another model.
Why it matters: These options address the prompt format and model capabilities identified in the diagnosis. - Customization and distribution paths: Distinguish the project’s two broader paths: use organization-specific code in Milvus and serve the API and database to colleagues, or use DeciCoder alone, potentially compiled to GGUF and run through a JavaScript SDK inside the extension. State why the current local setup is not yet ready to publish.
Why it matters: The choice changes how the copilot gets context and how it can be shared.
Key terms: RAG, DeciCoder, linked list, few-shot examples, HumanEval, instruction tuned, prompt tuning, example code, finetune, instruction datasets
10.5 Summary
The chapter summary recaps the coding copilot’s model, vector database, and editor extension, then emphasizes dataset preparation and prompts suited to the model.
Concepts explained in this chapter
- retrieval-augmented generationRetrieval-augmented generation is a technique that supplies a generative model with information retrieved from external sources at response time.
- VS Code extensionA VS Code extension is a packaged software component that adds capabilities to the Visual Studio Code editor.
- VS Code APIThe VS Code API is the programming interface through which extensions interact with Visual Studio Code and its editor features.
- VS Code extension template generatorA VS Code extension template generator is a tool that creates starter project files and scaffolding for a Visual Studio Code extension.
- coding copilotA coding copilot is a software assistant that generates or suggests source code in response to a developer's context or request.
- extension manifestAn extension manifest is a configuration document that declares an editor extension's identity, capabilities, activation behavior, and contributions.
- ghost textGhost text is provisional text rendered inline in an editor as a preview of a suggested completion before the user accepts it.
- inline completion itemAn inline completion item is a proposed text completion that a VS Code inline suggestion provider returns for display in the editor.
- inline suggestion providerAn inline suggestion provider is a VS Code extension component that supplies completion suggestions displayed at the cursor position.
- package.jsonpackage.json is a JSON configuration file that declares metadata, dependencies, scripts, and other package information for a JavaScript project.
- DeciCoderDeciCoder is a language model designed to generate source code, including Python, JavaScript, and Java code.
- fine-tuningFine-tuning is the process of continuing model training on task-specific data to adapt a pretrained model to a particular behavior or domain.
- vector databaseA vector database is a database optimized for storing and retrieving numerical vector representations using similarity-based search.
- context windowContext length is the maximum amount of tokenized input and generated context that a language model can process in one request.
- vectorizationVectorization is the transformation of input data into numerical vector representations.
- prompt engineeringPrompt engineering is the practice of designing and refining input instructions to influence an AI model's behavior and output.
- datasetA dataset is an organized collection of data used for analysis, evaluation, or machine-learning development.
- FastAPIFastAPI is a Python web framework for building HTTP APIs with type-based request validation and automatic interface documentation.
- promptA persistent prompt instruction applied to requests to establish consistent model behavior or interaction style.
- similarity searchSemantic search retrieves information according to the meaning represented by queries and documents rather than relying only on exact lexical matches.
- Visual Studio CodeVS Code is a source-code editor from Microsoft with extensibility for programming, debugging, and remote development.
- conversational aiAn AI assistant is software that interacts with users and performs tasks through artificial-intelligence capabilities.
- llm serviceAn LLM service is a deployed software service that exposes a large language model for remote requests and response generation.
- open-source modelA machine-learning model whose weights, code, or other defining artifacts are made available under terms permitting specified use, modification, or redistribution.
- few-shot learningFew-shot learning is the adaptation or prediction of a model from only a small number of labeled examples.
- infrastructureInfrastructure is the hardware, software, networking, and operational resources that support computing systems.
← Chapter 9: Creating an LLM project: Reimplementing Llama 3
Chapter 11: Deploying an LLM on a Raspberry Pi: How low can you go? →
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.