liveBook · Manning

Inference sampling

As explained in Domain-Specific Small Language Models

The process of generating tokens from a Transformer during inference, including prefill and autoregressive decoding.

Inference sampling generates tokens in two phases: prefill processes prompt tokens in parallel, then autoregressive decoding samples one token at a time. section 4.3.1

The main operational trade-offs are:

  • Prefill populates the key–value (KV) cache without generating tokens. section 4.3.1
  • Autoregressive decoding reuses cached attention state instead of recomputing all prior tokens. section 4.3.1
  • Caching consumes GPU VRAM or CPU RAM but can substantially reduce sampling time. section 4.3.1
  • Performance analysis compares arithmetic intensity with hardware memory-transfer and compute limits. section 4.3.1

Measuring KV-cache impact

A GPT-Neo generation benchmark on a T4 GPU compares Transformers generate calls with use_cache=True and use_cache=False; caching averages 10.281 seconds versus 51.978 seconds without it. section 4.3.1

How Domain-Specific Small Language Models covers it

How deep it goes
Explains prefill, decoding, KV caching, attention cost, arithmetic intensity, and a measured cache-speed comparison.
Examples
Uses GPT-Neo generation on a T4 GPU and compares Transformers generate calls with caching enabled or disabled.
What it assumes
Assumes familiarity with Transformer attention, token generation, GPU memory, and model-generation APIs.
What you can do after
Operate/configure generation with KV caching and diagnose whether inference is memory- or compute-bound.
How it connects
Connects Transformer inference with attention, autoregressive decoding, KV caching, and arithmetic intensity.
Cover of Domain-Specific Small Language Models
liveBookmanning.com
Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.