How deeply does Domain-Specific Small Language Models cover arithmetic intensity?

Defines the ratio, compares it to hardware capacity, and works through a Transformer attention calculation.

Arithmetic intensity

As explained in Domain-Specific Small Language Models

The number of compute operations divided by the number of bytes moved to and from memory for an algorithm or model layer.

Arithmetic intensity measures compute work per byte moved to and from memory. It helps identify whether a model layer is memory bound or compute bound on a given GPU. section 4.3.1 section 4.4

It is used as a hardware-agnostic way to compare an algorithm’s demand against memory traffic, then decide where optimization should focus. In Transformer inference, attention layers are highlighted because they are computationally intensive. section 4.3.1

A practical reading of the metric follows this sequence:

Compute arithmetic intensity as total compute divided by total memory movement. section 4.3.1
Compare the result with the hardware’s ops-per-byte capacity to classify the bottleneck. section 4.3.1
Use the bottleneck diagnosis to target either memory traffic or compute utilization. section 4.3.1

Transformer inference with KV cache

Sampling is split into prefill and autoregressive phases. KV caching stores attention state so later tokens avoid recomputing prior attention, and tests show a large speedup when caching is enabled. A worked attention example computes total memory movement, total compute, and an arithmetic intensity of 30.29 ops/byte. section 4.3.1 section 4.4

How Domain-Specific Small Language Models covers it

Cover of Domain-Specific Small Language Models

liveBook manning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

tilingproduced by · Tiling is a technique that partitions data or computation into smaller blocks to improve memory locality, data reuse, and processing efficiency.
compute-boundcontrasts with · A computation is compute-bound when its performance is limited primarily by available processor or arithmetic capacity rather than memory transfer, input/output, or other resources.
memory boundcontrasts with · A memory-bound workload is one whose performance is limited primarily by the rate of accessing or transferring data rather than by computational throughput.
kernel fusionproduced by · Kernel fusion is an optimization that combines multiple computational operations into fewer executable units to reduce launch overhead and intermediate data movement.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.