liveBook · Manning

Stride

As explained in GPU Programming with Triton

The number of storage elements between successive logical elements along an array or tensor dimension.

A stride is the number of storage elements between successive logical elements along one tensor dimension. Strides let a kernel translate multidimensional indices into offsets in linear memory, even when the tensor is a transposed or sliced view. section 5.2 section 5.2.1

For index (i0, i1, …) and strides (s0, s1, …), the element offset from the base pointer is i0·s0 + i1·s1 + …. Increasing one index by one changes the offset by that dimension’s stride. Shape describes the logical extent; strides describe how to reach its elements. section 5.2 section 5.2.1

A 3 × 4 matrix

A contiguous row-major matrix has strides (4, 1): moving down one row advances four elements, and moving right advances one. Element (2, 1) is at offset 2·4 + 1·1 = 9. For a column-major layout with strides (1, 3), the same logical position is at offset 5. section 5.2.1

Figure 5.2 Visualizing the stride formula. The logical position a 21 maps to physical offset 9 in a row-major layout with strides (4, 1), but to physical offset 5 in a column-major layout with strides (1, 3).
CH05 F02 Fartale

Compare the same logical index under row-major and column-major stride rules. section 5.2.1

A view can change the mapping without copying storage. Transposing the row-major 3 × 4 matrix produces a 4 × 3 view with strides (1, 4). Original index (1, 2) and transposed index (2, 1) both reach offset 6. If a subsequent operation requires contiguous row-major storage, making the view contiguous copies it into a new buffer. section 5.2.2

Two stride values are especially useful when inspecting a layout:

  • Stride 1 means neighboring elements along that dimension occupy consecutive storage locations. Neighboring GPU lanes reading along that dimension can coalesce their loads. section 5.2.3 section 5.10
  • Stride 0 means changing that index leaves the address unchanged. A broadcast view of a vector across rows can have strides (0, 1), so every row reads the same vector storage. section 5.2.3

In Triton, pass runtime strides with the shape and use them in pointer arithmetic rather than inferring layout from shape. Work through offsets for concrete dimensions before launching: check bounds, ensure outputs are covered, and verify that instances do not write the same location unless intended. section 5.2.3 section 3.5.4

Different row strides in a fused kernel

For a row-major input x shaped [seq_len, feature_dim] and cosine and sine tensors shaped [seq_len, feature_dim // 2], the row strides differ. A program handling row pid starts x at pid·feature_dim and the angle tensors at pid·(feature_dim // 2). Mask pair indices beyond the half dimension, and guard row IDs if the launch can exceed seq_len. section 3.5.3

Correct offsets do not guarantee fast access. Strided requests from neighboring lanes can transfer more unused data than consecutive requests. Choose the access direction with the physical layout in mind, especially for operations such as transpose. section 5.10

How GPU Programming with Triton covers it

How deep it goes
Derives the multidimensional offset formula and explains views, unit and zero strides, runtime indexing, and coalescing.
Examples
Works through matrix offsets, a shared-storage transpose, and distinct input and angle row strides in a fused RoPE kernel.
What you can do after
Apply it via Triton by passing runtime strides, calculating offsets, and checking bounds and write coverage.
How it connects
Connects tensor layout to transposition, slicing, broadcasting, and memory coalescing.

Also explained in

Compare how 11 books explain stride →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.