liveBook · Manning

Row-major order

As explained in GPU Programming with Triton

Row-major order is a memory layout in which the elements of each row are stored contiguously before the next row.

Row-major order stores a matrix one row at a time, making neighboring elements within each row adjacent in memory. For a contiguous matrix with N columns, element (i, j) is at flat offset i × N + j. section 5.1

A 3 × 4 matrix

Rows occupy offsets 0–3, 4–7, and 8–11. The element at (2, 1) is at offset 2 × 4 + 1 = 9; the corresponding strides are (4, 1). section 5.2.1

In a GPU kernel, row-major layout makes row starts easy to calculate: row i begins at i × N. But shape alone may not describe a tensor’s physical layout; transposes and slices can change its strides, so kernels may need runtime stride values. section 3.5.3 section 5.2.1

Figure 5.1 Row-major versus column-major memory layout. In row-major order (top), elements in the same row are adjacent in memory. In column-major order (bottom), elements in the same column are adjacent.
CH05 F01 Fartale

Compare which matrix direction is contiguous in row-major and column-major layouts. section 5.1

How GPU Programming with Triton covers it

How deep it goes
Defines contiguous row storage and works through flat-offset and stride calculations, including noncontiguous tensor views.
Examples
Works through a 3 × 4 matrix and calculates the flat offset and strides for element (2, 1).
What you can do after
Apply the flat-index formula to locate matrix elements in contiguous row-major storage.
How it connects
Connects row-major order to column-major order, strides, and memory coalescing.

Also explained in

Compare how 2 books explain row-major order →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.