Row-major order
Row-major order is a multidimensional array layout in which the rightmost index varies fastest, so the elements of each row are contiguous in memory.
GPU Programming with Triton
Row-major order is a memory layout in which the elements of each row are stored contiguously before the next row.
- How it is used
- Uses row-major layout to calculate matrix addresses and reason about GPU memory access.
- How deep it goes
- Defines contiguous row storage and works through flat-offset and stride calculations, including noncontiguous tensor views.
- Examples
- Works through a 3 × 4 matrix and calculates the flat offset and strides for element (2, 1).
- What you can do after
- Apply the flat-index formula to locate matrix elements in contiguous row-major storage.
Read how GPU Programming with Triton explains row-major order →
Grokking GPU Programming
Row-major order is a multidimensional array layout in which elements of each row are contiguous in memory.
- How it is used
- Uses row-major layout to reason about matrix indexing, GPU memory access, and library interoperability.
- How deep it goes
- Works through flattening, indexing errors, warp access patterns, shared-memory padding, and the cuBLAS transpose trick.
- Examples
- Uses a 2×3 matrix, row and column sums, padded shared-memory tiles, and a cuBLAS matrix multiplication call.
- What you can do after
- Diagnose/debug indexing and access-pattern problems using row-major offsets and small hand-checked matrices.
Read how Grokking GPU Programming explains row-major order →
Also covered in
- Learn Go with Pocket-Sized Projects
- Transformers in Action
- Fast Python: High performance techniques for large datasets
- Deep Learning with R, Third Edition
What to know first
- arrayAn array is an ordered collection of elements organized in one or more dimensions and accessible by position or index.
- C++C++ is a statically typed, general-purpose programming language that supports procedural, object-oriented, generic, and low-level systems programming.
- shared memoryShared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data.
Related concepts
- column-major ordercontrasts with · Column-major order is a multidimensional array layout in which the leftmost dimension varies fastest, making each matrix column contiguous in memory.
- PyTorchconfigured by · PyTorch is an open-source machine-learning framework that provides tensor computation, automatic differentiation, and tools for developing and training neural networks.
- memory layoutkind of · Memory layout is the arrangement and mapping of data elements or regions to locations in memory.
- arrayconfigures · An array is an ordered collection of elements organized in one or more dimensions and accessible by position or index.
Questions about row-major order
What is row-major order?
Row-major order is a multidimensional array layout in which the rightmost index varies fastest, so the elements of each row are contiguous in memory.
Which Manning books explain row-major order?
GPU Programming with Triton and Grokking GPU Programming.
What should I know before learning row-major order?
Manning books that teach row-major order usually introduce array, C++, Fortran and shared memory first.
What is the difference between row-major order and column-major order?
Row-major order is a multidimensional array layout in which the rightmost index varies fastest, so the elements of each row are contiguous in memory. Column-major order is a multidimensional array layout in which the leftmost dimension varies fastest, making each matrix column contiguous in memory.
What is row-major order a kind of?
Row-major order is a kind of memory layout. Memory layout is the arrangement and mapping of data elements or regions to locations in memory.
What is row-major order also called?
Row-major order is also called row-major layout.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

