liveBook · Manning

StarCoder2

As explained in Domain-Specific Small Language Models

A code-generation language model family used here as a custom model example, with the 3B variant specifically mentioned.

StarCoder2 is a 3-billion-parameter code-generation model designed for practical local inference. It was trained on more than 3 trillion tokens spanning 17 programming languages and curated natural-language sources. Grouped-query attention speeds inference, while its 16,384-token context includes 4,096-token sliding-window attention. section 7.2.4

Use StarCoder2 with these operational boundaries:

  • Prompt a function header, comment, or docstring when generating code; because the model is not instruction-tuned, function-body completion is its strongest stated use. section 7.2.4
  • Load the model through Hugging Face Transformers, then generate with AutoModelForCausalLM or a text-generation pipeline. section 7.2.4
  • Reduce memory requirements with Bitsandbytes 8-bit or 4-bit quantization; 4-bit loading reduces model size by about 75% while preserving acceptable quality. section 7.2.4 section 7.4
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "bigcode/starcoder2-3b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id,
                                             device_map='auto',
                                             torch_dtype=torch.bfloat16)
model.eval()

Shows the practical loading pattern for the StarCoder2-3B checkpoint with automatic device placement and bfloat16 inference. section 7.2.4

How Domain-Specific Small Language Models covers it

How deep it goes
Details training data, attention design, context limits, prompting boundaries, loading, pipelines, and quantization, but not tuning procedures.
Examples
Uses function headers such as fibonacci, Transformers pipelines, and 8-bit or 4-bit Bitsandbytes loading.
What it assumes
Assumes Python code prompts and teaches model, tokenizer, pipeline, and quantized-checkpoint loading from the beginning.
What you can do after
Operate StarCoder2 through Transformers, generate code from prompts, and configure lower-memory inference.
How it connects
Connects grouped-query attention, code infilling, Hugging Face Transformers, quantization, and syntactic validation.
Cover of Domain-Specific Small Language Models
liveBookmanning.com
Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.