liveBook · Manning

Effective memory bandwidth

As explained in GPU Programming with Triton

The achieved rate of data transfer during a computation, as distinct from the memory system's peak bandwidth.

Effective memory bandwidth is the achieved rate of data transfer during a computation, rather than the memory system’s peak bandwidth. It is useful for memory-bound workloads, where moving data costs much more than the arithmetic. section 2.2.10

For vector addition, each element requires two reads and one write. With float32 inputs and output, the workload moves 12 bytes per element, so for N elements and an elapsed time of t seconds, effective bandwidth is 12N / (t × 10⁹) GB/s. This calculation uses the workload’s expected data transfer and the measured execution time. section 2.2.10

A useful measurement needs representative timing. Check correctness, exclude first-call cost, warm up and repeat measurements, and control cache effects. Triton’s do_bench uses GPU events for GPU work; a CPU Python loop instead needs wall-clock timing because it queues no GPU work between those events. section 2.2.10

Comparing vector additions

A size sweep compares a Python CPU loop, PyTorch GPU addition, and a Triton GPU kernel. As vectors grow, the Python loop’s throughput falls close to zero, while PyTorch and Triton scale toward the GPU’s memory-bandwidth limits and perform comparably. section 2.2.10

Access patterns can also reduce effective bandwidth within shared memory. In a 32 × 32 tile, a column read can put all 32 lanes on the same bank, splitting one request into 32. Padding the tile to 32 × 33 distributes that column across the banks. This calculation does not imply that a Triton transpose kernel necessarily creates a padded physical buffer. section 5.4.4

Figure 5.10 Bank mapping for a 32 × 32 shared-memory tile and a padded 32 × 33 tile. Without padding, every 32-bit value in a column maps to the same bank, and the warp request is split into 32 conflict-free requests. Padding changes the row stride to 33 values, distributing the column across all 32 banks in one conflict-free request.
CH05 F10 Fartale

Trace the bank mapping before and after padding a shared-memory tile. section 5.4.4

How GPU Programming with Triton covers it

How deep it goes
Explains the vector-addition metric and measurement controls, then relates shared-memory bandwidth loss to bank conflicts.
Examples
Compares Python, PyTorch, and Triton vector additions and examines padded versus unpadded shared-memory transpose tiles.
What you can do after
Evaluate vector-addition performance using effective bandwidth and controlled timing, and understand how bank conflicts can reduce transpose bandwidth.
How it connects
Connects effective bandwidth to memory-bound workloads, caching, GPU timing, shared memory, and bank conflicts.

Also explained in

Compare how 2 books explain effective memory bandwidth →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.