liveBook · Manning

Memory layout

As explained in GPU Programming with Triton

Memory layout is the mapping between logical data structures and their locations in memory.

A matrix may be two-dimensional logically, but its data occupies a linear sequence of memory locations. Memory layout maps each logical index to a physical location. That mapping determines both which values a kernel reads and how efficiently it can move them. section 5.1 section 5.10

For a tensor with indices (i0, …, in−1) and element strides (s0, …, sn−1), the offset from the data buffer is i0×s0 + … + in−1×sn−1. Each stride is the number of elements advanced when its index increases by one. Adding the offset to the base pointer identifies the requested element. section 5.2

Two common contiguous matrix layouts differ in which direction is adjacent in memory:

  • Row-major order stores each row consecutively. For a matrix with N columns, element (i, j) has offset i×N + j. section 5.1
  • Column-major order stores each column consecutively, making elements in the same column adjacent. section 5.1
Figure 5.1 Row-major versus column-major memory layout. In row-major order (top), elements in the same row are adjacent in memory. In column-major order (bottom), elements in the same column are adjacent.
CH05 F01 Fartale

Compare how the same logical matrix maps to consecutive locations in row-major and column-major order. section 5.1

Shape alone does not determine the mapping. Views, slices, transposes, and broadcasting can change how indices address existing storage without copying it. Passing strides to a kernel lets its address calculations follow the actual layout rather than assume a fixed order. section 5.10 section 5.5.2

Rotation with split feature pairs

For a row of width D, a split layout pairs feature i with feature i+D/2. With shared rotation frequencies, the cosine and sine tables have row width D/2. A kernel must use the appropriate row width for each input: an offset error can load the wrong features or rotation values. section 3.5.2

Correct addresses are necessary but not sufficient for speed. On a GPU, memory coalescing depends on addresses requested together by neighboring lanes in a warp. Consecutive addresses use fewer global-memory transactions than strided accesses, which transfer more unused data. A tiled transpose can stage reordering on chip to improve access patterns; boundary masks protect loads and stores at incomplete tiles. section 5.10

Matrix multiplication access

In a naive matrix multiplication kernel with row-major inputs, a strip from a row of A advances through adjacent elements, while a strip down a column of B advances by its row stride. Each output program also loads its inputs independently. Tiling allows loaded values to contribute to multiple outputs, improving reuse. section 5.5.2 section 5.10

The level of layout control depends on the programming tool. Triton lets the programmer choose tile shapes and grid division while the compiler makes other decisions; Gluon exposes more direct control over tile arrangement and scheduling, at a portability cost across GPU generations. section 1.2.2

How GPU Programming with Triton covers it

How deep it goes
Explains stride-based address calculation, layout-dependent access patterns, and tiled kernel implications.
Examples
Works through row-major and column-major matrices, split-pair rotations, and a naive matrix multiplication kernel.
What you can do after
Apply stride-based indexing in Triton kernels and evaluate the access and reuse consequences of different layouts.
How it connects
Connects tensor strides, broadcasting, memory coalescing, matrix transpose, and matrix multiplication.

Also explained in

Compare how 5 books explain memory layout →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.