liveBook · Manning

Column-major order

Column-major order is a multidimensional array layout in which the leftmost dimension varies fastest, making each matrix column contiguous in memory.

How 3 Manning books explain it

GPU Programming with Triton

Column-major order is a memory layout in which the elements of each column are stored contiguously before the next column.

How it is used
Uses column-major order to explain matrix addressing and memory access patterns in higher-dimensional GPU kernels.
How deep it goes
Defines the layout, gives its flat-index formula, and connects contiguous access to memory behavior.
Examples
Uses a 3 × 4 matrix, its flat storage sequence, and language conventions to illustrate the layout.
What you can do after
Understand how to map matrix coordinates to flat offsets and assess which direction accesses contiguous memory.

Read how GPU Programming with Triton explains column-major order →

Grokking GPU Programming

Column-major order is a multidimensional array layout in which elements of each column are contiguous in memory.

How it is used
Distinguishes matrix storage conventions when reasoning about CUDA memory access and library interoperability.
How deep it goes
Defines column-wise contiguity, contrasts indexing conventions, and connects traversal direction to warp access patterns.
Examples
Uses a 2×3 matrix and contrasts CUDA C++ indexing with cuBLAS’s column-major interpretation.
What you can do after
Evaluate matrix indexing and library layout assumptions to avoid transpose-like results or misread data.

Read how Grokking GPU Programming explains column-major order →

CUDA for LLMs

A column-major layout stores multidimensional array elements so that the leftmost dimension varies fastest in linear memory.

How it is used
Handles matrix layout conventions when calling cuBLAS matrix multiplication from C or C++ code.
How deep it goes
Defines the leading dimension and explains cuBLAS dimension adjustments for row-major inputs.
Examples
Uses cublasSgemm and the B @ A reversal with swapped m and n to handle row-major data.
What you can do after
Apply the row-major adaptation by reversing multiplication order and swapping the m and n dimensions.

Read how CUDA for LLMs explains column-major order →

Also covered in

What to know first

Related concepts

Questions about column-major order

What is column-major order?

Column-major order is a multidimensional array layout in which the leftmost dimension varies fastest, making each matrix column contiguous in memory.

Which Manning books explain column-major order?

GPU Programming with Triton, Grokking GPU Programming and CUDA for LLMs.

What should I know before learning column-major order?

Manning books that teach column-major order usually introduce C++, Fortran, CUDA and float16 first.

What is the difference between column-major order and row-major order?

Column-major order is a multidimensional array layout in which the leftmost dimension varies fastest, making each matrix column contiguous in memory. Row-major order is a multidimensional array layout in which the rightmost index varies fastest, so the elements of each row are contiguous in memory.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.