Chapter 7: Generating Python code
The chapter introduces Python code generation as a way to examine specialized open-source LLMs and judge their outputs.
After this chapter you will be able to
- Python as an evaluation language. Explain why the chapter uses Python even though most open-source code-generation models support multiple languages.
- Limits of generalist coding models. Explain what real-world programming benchmarks indicate about generalist models and why software engineering problems are harder than straightforward code generation.
- Benefits of focused coding models. Explain why specialization can make smaller coding models competitive and their training data easier to manage.
- Human role in software development. Explain why highly accurate code generation would not, by itself, replace software engineers and who benefits most from a code assistant.
- CodeGen prompting options. Describe how a prompt can elicit a Python function body from CodeGen and how the prompting options differ by model size.
- ONNX optimization workflow. Trace the section’s workflow for converting CodeGen 350M mono to ONNX and then dynamically quantizing it.
- CPU benchmark findings. Explain how the three CodeGen versions were benchmarked and what changes the section reports in inference performance.
- ONNX conversion without Optimum. Describe the conversion path for a model architecture unsupported by Optimum: save the downloaded model locally, run the Transformers library’s transformers.onnx script, and set its feature and model options.
- HumanEval evaluation inputs. Describe what HumanEval supplies for evaluating generated Python code and how its prompts enter the generation workflow.
- AST syntax validation. Explain how the workflow uses an abstract syntax tree to filter generated code before testing it.
- Correctness testing and safety. Describe how valid generated code is checked for correctness and why running that evaluation calls for caution.
- CodeGen generation and infilling. Describe the CodeGen 2.5 mono generation workflow and the required prompt markers for infilling.
- StarCoder2 completion workflow. Explain what kind of prompt suits StarCoder2 and how the section generates code with it.
- GGUF runtime choice. Explain why downloading a GGUF checkpoint through Transformers does not provide quantized local inference, and why llama.cpp is used instead.
- Local code generation workflow. Describe how the example downloads, loads, and prompts the quantized StarCoder model for CPU-only code generation, including how to obtain just the generated text.
What it builds on
- Code generation (chapter 4) · The earlier discussion of code generation.
- Domain-specific vs. generalist LLMs for business value (chapter 1) · The distinction between domain-specific and generalist language models.
- 8-bit quantization with ONNX (chapter 6) · ONNX quantization
- Evaluating the generated content (chapter 4) · evaluating generated content as distinct from measuring inference performance
- 8-bit quantization (chapter 6) · Assumes the earlier treatment of 8-bit quantization.
- 4-bit quantization (chapter 6) · Assumes the earlier treatment of 4-bit quantization.
- 4-bit quantization with ggml (chapter 6) · The earlier discussion of quantized models and ggml.
Section by section
Introduction
The chapter introduces Python code generation as a way to examine specialized open-source LLMs and judge their outputs.
- Python as an evaluation language: Explain why the chapter uses Python even though most open-source code-generation models support multiple languages.
Why it matters: The choice of Python makes the models’ outputs easier to assess.
Key terms: open source models, Python, quality of the outputs
7.1 Using Transformers to generate code
This section explains why coding assistants remain tools for supervised software engineering and why smaller, specialized coding models are promising alternatives to large generalist models.
- Limits of generalist coding models: Explain what real-world programming benchmarks indicate about generalist models and why software engineering problems are harder than straightforward code generation.
Why it matters: A strong-looking code suggestion is not the same as a reliable solution to a real software issue. - Dedicated assistant tradeoffs: Distinguish the potential coding advantages of dedicated assistants from their organizational costs and risks.
Why it matters: Choosing a coding assistant involves more than comparing how well it writes code. - Benefits of focused coding models: Explain why specialization can make smaller coding models competitive and their training data easier to manage.
Why it matters: A focused model can be a practical alternative when a large general-purpose model carries unnecessary size and data costs. - Human role in software development: Explain why highly accurate code generation would not, by itself, replace software engineers and who benefits most from a code assistant.
Why it matters: It sets realistic expectations for what an AI coding assistant can contribute to a development team.
Key terms: SWE-bench, generalist models, expert human supervision, dedicated models, human oversight, guardrails, data leakage, specialization, coding LLMs, focused data pool
7.2 Generating Python code with a Transformer architecture
7.2.1 Python code generation with CodeGen
This section uses CodeGen 350M mono to demonstrate Python code prompting and to measure how ONNX conversion and 8-bit quantization affect CPU inference.
- CodeGen model selection: Distinguish CodeGen’s mono and multi families and explain why the section uses 350M mono despite its production limitations.
Why it matters: It clarifies why the optimization example uses a small Python-focused model without treating its output quality as production-ready. - CodeGen prompting options: Describe how a prompt can elicit a Python function body from CodeGen and how the prompting options differ by model size.
Why it matters: A reader needs to match the prompt style to the CodeGen model being used. - ONNX optimization workflow: Trace the section’s workflow for converting CodeGen 350M mono to ONNX and then dynamically quantizing it.
Why it matters: This workflow produces the model versions whose CPU inference performance the section compares. - CPU benchmark findings: Explain how the three CodeGen versions were benchmarked and what changes the section reports in inference performance.
Why it matters: The measurements show that optimizing even a small model can improve CPU inference, while leaving generated-code quality as a separate question.
Key terms: CodeGen, mono, multi, 350M, prompt, function definition, natural-language prompts, inline comment, unit tests, Optimum
7.2.2 Using ONNX with models not supported by Optimum
This section explains how to convert a model to ONNX when Optimum does not support its architecture, before using ONNX Runtime for 8-bit quantization.
- ONNX conversion without Optimum: Describe the conversion path for a model architecture unsupported by Optimum: save the downloaded model locally, run the Transformers library’s transformers.onnx script, and set its feature and model options.
Why it matters: An unsupported Optimum architecture does not prevent you from preparing the model for ONNX-based quantization.
Key terms: Optimum, mid-level approach, transformers.onnx, --feature, causal-lm, --model
7.2.3 Model evaluation
This section explains a workflow for evaluating Python code generated by CodeGen: prepare HumanEval prompts, filter invalid Python with an AST, and run tests on valid outputs, while noting robustness and safety considerations.
- HumanEval evaluation inputs: Describe what HumanEval supplies for evaluating generated Python code and how its prompts enter the generation workflow.
Why it matters: The evaluation needs both a prompt to generate from and a test to check the result. - AST syntax validation: Explain how the workflow uses an abstract syntax tree to filter generated code before testing it.
Why it matters: Filtering invalid Python prevents syntactically broken outputs from proceeding as testable code. - Correctness testing and safety: Describe how valid generated code is checked for correctness and why running that evaluation calls for caution.
Why it matters: Passing a syntax check is not the same as solving the problem, and testing generated code means executing it. - Robustness evaluation motivation: Explain why the section selects a ReCode-based approach for evaluating code generation models.
Why it matters: A code generator’s performance on one wording of a prompt may not reflect how reliably it works in production.
Key terms: HumanEval, prompt, test, CodeGen tokenizer, ast, abstract syntax tree (AST), valid Python, HumanEval test, passed, execution.py
7.2.4 Python code generation with better models
This section shows how to generate and infill Python code with CodeGen 2.5, introduces StarCoder2 as another code-generation model, and notes their deployment and licensing considerations.
- CodeGen 2.5 efficient attention: Explain why CodeGen 2.5 is suited to efficient serving and local deployment, including the role of FlashAttention.
Why it matters: These features help explain why the model is practical to serve or run locally. - CodeGen 2.5 license distinction: Distinguish the licensing of CodeGen 2.5 mono and multi models from that of its instruct model.
Why it matters: The model variant matters when deciding whether it can be used commercially. - CodeGen generation and infilling: Describe the CodeGen 2.5 mono generation workflow and the required prompt markers for infilling.
Why it matters: The prompt format determines whether the model can fill a gap in an existing Python snippet. - StarCoder2 completion workflow: Explain what kind of prompt suits StarCoder2 and how the section generates code with it.
Why it matters: A function header gives this non-instruction-tuned model a suitable starting point for code completion. - StarCoder2 quantization and performance: Describe the quantization options shown for StarCoder2 and the reported GPU benchmark comparison.
Why it matters: These options and measurements help compare code-generation models under hardware constraints.
Key terms: CodeGen 2.5, FlashAttention, infill sampling, GPU HBM, SRAM, Apache 2.0, CodeGen 2.5 instruct, mono, multi, CodeGen 2.5 mono
7.3 Coding assistance on commodity hardware
This section shows how to run a quantized code-generation model as a local coding assistant on a modest laptop, and how to prepare a GGUF model when one is not already suitable.
- GGUF runtime choice: Explain why downloading a GGUF checkpoint through Transformers does not provide quantized local inference, and why llama.cpp is used instead.
Why it matters: Choosing a runtime that actually uses the quantized model is what makes inference on a memory-limited laptop practical. - Local code generation workflow: Describe how the example downloads, loads, and prompts the quantized StarCoder model for CPU-only code generation, including how to obtain just the generated text.
Why it matters: This is the section’s concrete path from a quantized checkpoint to an offline coding assistant. - Laptop performance and integration: Describe the section’s performance finding and the ways the local model can provide coding assistance beyond a Python script.
Why it matters: The example connects workable laptop performance to practical code-completion and suggestion workflows. - Custom GGUF preparation: Describe what to do when the desired model is not available in a suitable GGUF version and what inputs the conversion script needs.
Why it matters: A ready-made GGUF checkpoint is not required to try the same local-inference approach with another model.
Key terms: GGUF, Transformers, llama.cpp, quantized, llama-cpp-python, Llama, StarCoder v2 3B, 4-bit K_M, context size, GPU layers
7.4 Summary
Concepts explained in this chapter
- CodeGenA pretrained code-generation transformer model family used for Python code generation and inference optimization.
- CPUExecutionProviderThe ONNX Runtime execution provider that runs model computation on CPU-backed execution.
- CodeGenForCausalLMThe Hugging Face auto class specialized for loading CodeGen causal language models.
- ORTModelForCausalLMThe Optimum ONNX Runtime class for loading a causal language model in ONNX Runtime format.
- OptimumA Hugging Face extension library for optimizing transformer models for training and inference on target hardware, including ONNX workflows.
- ReCodeAn evaluation approach for code generation models that measures robustness under prompt perturbations.
- coding assistantA model-based assistant intended to help write or suggest program code in an editor or chat workflow.
- StarCoder2A code-generation language model family used here as a custom model example, with the 3B variant specifically mentioned.
- grouped-query attentionAn attention variant that speeds inference by sharing attention computation across groups of queries.
- int4 quantizationA quantization method that represents model weights or activations with 8-bit numbers to reduce model size and improve efficiency.
- quantizationMethods that reduce numeric precision or representational size of model values to improve efficiency, especially for inference; here the absmax method is a specific quantization procedure.
- ONNX RuntimeA runtime library that efficiently executes ONNX models across hardware targets such as CPUs, GPUs, and edge devices.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.