Column-major order
Column-major order is a multidimensional array layout in which the leftmost dimension varies fastest, making each matrix column contiguous in memory.
GPU Programming with Triton
Column-major order is a memory layout in which the elements of each column are stored contiguously before the next column.
- How it is used
- Uses column-major order to explain matrix addressing and memory access patterns in higher-dimensional GPU kernels.
- How deep it goes
- Defines the layout, gives its flat-index formula, and connects contiguous access to memory behavior.
- Examples
- Uses a 3 × 4 matrix, its flat storage sequence, and language conventions to illustrate the layout.
- What you can do after
- Understand how to map matrix coordinates to flat offsets and assess which direction accesses contiguous memory.
Read how GPU Programming with Triton explains column-major order →
Grokking GPU Programming
Column-major order is a multidimensional array layout in which elements of each column are contiguous in memory.
- How it is used
- Distinguishes matrix storage conventions when reasoning about CUDA memory access and library interoperability.
- How deep it goes
- Defines column-wise contiguity, contrasts indexing conventions, and connects traversal direction to warp access patterns.
- Examples
- Uses a 2×3 matrix and contrasts CUDA C++ indexing with cuBLAS’s column-major interpretation.
- What you can do after
- Evaluate matrix indexing and library layout assumptions to avoid transpose-like results or misread data.
Read how Grokking GPU Programming explains column-major order →
CUDA for LLMs
A column-major layout stores multidimensional array elements so that the leftmost dimension varies fastest in linear memory.
- How it is used
- Handles matrix layout conventions when calling cuBLAS matrix multiplication from C or C++ code.
- How deep it goes
- Defines the leading dimension and explains cuBLAS dimension adjustments for row-major inputs.
- Examples
- Uses cublasSgemm and the B @ A reversal with swapped m and n to handle row-major data.
- What you can do after
- Apply the row-major adaptation by reversing multiplication order and swapping the m and n dimensions.
Also covered in
What to know first
- C++C++ is a statically typed, general-purpose programming language that supports procedural, object-oriented, generic, and low-level systems programming.
- CUDACUDA is NVIDIA’s parallel computing platform and programming model for performing general-purpose computations on NVIDIA GPUs.
- float16float16 is a 16-bit floating-point format, commonly IEEE 754 binary16, with less precision and range than 32-bit floating-point formats.
Related concepts
- row-major ordercontrasts with · Row-major order is a multidimensional array layout in which the rightmost index varies fastest, so the elements of each row are contiguous in memory.
- cuBLASconfigured by · cuBLAS is NVIDIA’s GPU-accelerated library of basic linear algebra operations for CUDA applications.
- WGMMAused by · WGMMA is an NVIDIA GPU instruction family that performs matrix multiply-accumulate operations collectively across a warp group.
- transpositionproduces · Transposition is an operation that exchanges a matrix’s rows and columns or, more generally, permutes the axes of a multidimensional array.
Questions about column-major order
What is column-major order?
Column-major order is a multidimensional array layout in which the leftmost dimension varies fastest, making each matrix column contiguous in memory.
Which Manning books explain column-major order?
GPU Programming with Triton, Grokking GPU Programming and CUDA for LLMs.
What should I know before learning column-major order?
Manning books that teach column-major order usually introduce C++, Fortran, CUDA and float16 first.
What is the difference between column-major order and row-major order?
Column-major order is a multidimensional array layout in which the leftmost dimension varies fastest, making each matrix column contiguous in memory. Row-major order is a multidimensional array layout in which the rightmost index varies fastest, so the elements of each row are contiguous in memory.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


