liveBook · Manning

Warming up

As explained in GPU Programming with Triton

A warmup run is an untimed execution performed before measurement to complete initialization and bring hardware into a representative operating state.

A warmup run executes a kernel before timing begins. For steady-state measurements, discard the first call and usually several early calls so that one-time work does not distort the result. section 2.2.10

A first Triton call can include CPU compilation as well as GPU execution. Triton caches the compiled binary, so subsequent calls with the same specialization can skip compilation. Changing a constexpr value such as BLOCK_SIZE triggers a fresh compile; changing the runtime vector length does not directly do so. section 2.2.10

Warm up each tuning case

When comparing launch configurations, warm up before timing a new shape: the first launch for a new autotuning key includes the work of measuring candidate configurations. Later launches with the same key use the selected configuration. section 5.8.3

triton.testing.do_bench(
    fn,
    warmup=25,
    rep=100,
    grad_to_none=None,
    quantiles=None,
    return_mode='mean',
)

The Triton benchmarking helper exposes a warmup parameter alongside its measurement settings. section 2.2.10

How GPU Programming with Triton covers it

How deep it goes
Explains discarded early calls and how compilation caching and autotuning affect when warmup is needed.
Examples
Uses vector addition and autotuned launch configurations to illustrate first-call costs.
What you can do after
Evaluate steady-state kernel performance without including compilation or first-launch autotuning work.
How it connects
Connects warmup to compilation, specialization, caching, autotuning, and kernel benchmarking.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.