liveBook · Manning

Column-major order

As explained in Grokking GPU Programming

Column-major order is a multidimensional array layout in which elements of each column are contiguous in memory.

Column-major order stores a matrix one column at a time: elements in the same column are adjacent in memory. For a 2×3 matrix [[a, b, c], [d, e, f]], the memory sequence is [a, d, b, e, c, f]. section 10.10

Indexing the same matrix two ways

In column-major storage, an element at row row and column col is at col * number_of_rows + row. Using that formula on row-major data instead can produce a transpose-like result. For a row-major matrix, use row * number_of_columns + col. section 12.12 section 10.10

Figure 10.9 Column-major layout. The same matrix, with consecutive 1D indices now filling each column before moving to the next. Column 0 holds [0] through [n−1], so row 0 holds [0], [n], [2n], …, [(m−1)n]: stepping down one row moves one element in memory, while stepping right one column jumps n elements. Consecutive elements in memory are therefore consecutive rows, not consecutive columns. The numbered arrows reverse the order: down the rows first (1), then across to the next column (2).

Use the layout diagram to trace how consecutive memory indices move down a column before advancing to the next one. section 10.10

Layout is a convention chosen by the code or library, not by CUDA itself:

  • C/C++ matrices are typically row-major, while Fortran and cuBLAS default to column-major. section 10.10
  • cuBLAS can work with packed row-major matrices using a transpose-based call arrangement; padded buffers or submatrices require different leading dimensions. section 12.9

How Grokking GPU Programming covers it

How deep it goes
Defines column-wise contiguity, contrasts indexing conventions, and connects traversal direction to warp access patterns.
Examples
Uses a 2×3 matrix and contrasts CUDA C++ indexing with cuBLAS’s column-major interpretation.
What you can do after
Evaluate matrix indexing and library layout assumptions to avoid transpose-like results or misread data.
How it connects
Connects to row-major order, CUDA, Fortran, cuBLAS, and memory coalescing.

Also explained in

Compare how 3 books explain column-major order →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.