A stride is the number of storage elements between successive logical elements along one tensor dimension. Strides let a kernel translate multidimensional indices into offsets in linear memory, even when the tensor is a transposed or sliced view. section 5.2 section 5.2.1
For index (i0, i1, …) and strides (s0, s1, …), the element offset from the base pointer is i0·s0 + i1·s1 + …. Increasing one index by one changes the offset by that dimension’s stride. Shape describes the logical extent; strides describe how to reach its elements. section 5.2 section 5.2.1
A 3 × 4 matrix
A contiguous row-major matrix has strides (4, 1): moving down one row advances four elements, and moving right advances one. Element (2, 1) is at offset 2·4 + 1·1 = 9. For a column-major layout with strides (1, 3), the same logical position is at offset 5. section 5.2.1
Figure 5.2 Visualizing the stride formula. The logical position a 21 maps to physical offset 9 in a row-major layout with strides (4, 1), but to physical offset 5 in a column-major layout with strides (1, 3).
Compare the same logical index under row-major and column-major stride rules. section 5.2.1
A view can change the mapping without copying storage. Transposing the row-major 3 × 4 matrix produces a 4 × 3 view with strides (1, 4). Original index (1, 2) and transposed index (2, 1) both reach offset 6. If a subsequent operation requires contiguous row-major storage, making the view contiguous copies it into a new buffer. section 5.2.2
Two stride values are especially useful when inspecting a layout:
- Stride 1 means neighboring elements along that dimension occupy consecutive storage locations. Neighboring GPU lanes reading along that dimension can coalesce their loads. section 5.2.3 section 5.10
- Stride 0 means changing that index leaves the address unchanged. A broadcast view of a vector across rows can have strides (0, 1), so every row reads the same vector storage. section 5.2.3
In Triton, pass runtime strides with the shape and use them in pointer arithmetic rather than inferring layout from shape. Work through offsets for concrete dimensions before launching: check bounds, ensure outputs are covered, and verify that instances do not write the same location unless intended. section 5.2.3 section 3.5.4
Different row strides in a fused kernel
For a row-major input x shaped [seq_len, feature_dim] and cosine and sine tensors shaped [seq_len, feature_dim // 2], the row strides differ. A program handling row pid starts x at pid·feature_dim and the angle tensors at pid·(feature_dim // 2). Mask pair indices beyond the half dimension, and guard row IDs if the launch can exceed seq_len. section 3.5.3
Correct offsets do not guarantee fast access. Strided requests from neighboring lanes can transfer more unused data than consecutive requests. Choose the access direction with the physical layout in mind, especially for operations such as transpose. section 5.10










