liveBook · Manning

Stride

As explained in Grokking GPU Programming

Memory stride is the distance between memory locations accessed by successive elements of an access pattern.

If neighboring GPU threads read elements 0, 2, 4, and 6, they skip every other element. Their access has a stride of 2: stride is the distance between successive positions in an access pattern. A stride of 1 means consecutive elements. section 10.6

Stride can also mean the distance between values combined by one thread in an algorithm. That partner distance is different from the spacing between addresses accessed by neighboring threads. Check which distance is meant before judging an access pattern. section 5.8 section 11.7

Stride in a reduction and a scan

To reduce eight values to one sum, pair each active position with a partner 4 positions away, then 2, then 1. The active threads stay grouped at the front. A Hillis–Steele scan goes the other way: its partner distance doubles from 1 to 2 to 4 as each position gathers values from farther left. section 5.8 section 6.2

In the scan, a position without a left partner copies its value instead of adding. Each round reads from one buffer and writes to another, then the buffers swap; a barrier lets all threads finish before the next round. This prevents a thread from overwriting a value another still needs. section 6.2 section 6.12

Figure 6.1 The Hillis-Steele scan in 3 rounds, with double buffering. Each round doubles its stride (1, 2, 4) and writes results to the alternate buffer. Dotted boxes are cells left unchanged, dashed boxes are cells copied across, and solid boxes are newly computed sums. The pin (read source) and pout (write target) roles swap every round, so no thread overwrites a value another thread still needs. After round 3, buf[1] holds the final inclusive prefix sum.

Follow the scan’s doubling partner distances and alternating buffers across three rounds. section 6.2

For global memory, look at where all the threads in a warp access at once. Consecutive addresses can be coalesced; larger gaps can make the GPU fetch data no thread requested. In the aligned example below, stride-2 reads use only half the fetched bytes. Misalignment can require still more transactions. section 10.6 section 10.14

Figure 10.4 Strided access (stride 2): 32 threads, 2 transactions. The warp’s 32 threads (T0 … T31) each read every other memory element ([0], [2], …, [62]), so their addresses span 64 elements instead of 32. The two brackets below memory mark two 128-byte cache lines: T0 through T15 fall in the first, T16 through T31 in the second. Because the warp’s addresses straddle two cache lines, the hardware must issue two memory transactions instead of one, and only half of each transaction’s bytes are actually used. Following the Snapshot Rule, picture 32 arrows split across two cache lines, with every other slot empty. (In sectors: eight 32-byte sectors fetched for 128 useful bytes, the same 50%.)

See how 32 stride-2 reads span two cache lines rather than one in the illustrated model. section 10.6

Change the layout, not just the index

Suppose each particle stores six consecutive fields. When neighboring threads read the same field from neighboring particles, their reads are six elements apart. Storing each field in its own array makes those same-field reads consecutive. The best layout still depends on which fields the kernel uses. section 10.9

In shared memory, strided accesses can instead cause bank conflicts. For 4-byte floats in the illustrated 32-bank arrangement, stride 1 sends neighboring threads to different banks; stride 2 sends pairs of threads to the same banks, requiring two service rounds. section 11.7

A practical check is to write down the addresses neighboring threads access, then consider the memory layout and alignment. Separately, for a multi-round algorithm, check the partner distance, boundaries, and synchronization before optimizing. section 10.6 section 10.9 section 6.12

How Grokking GPU Programming covers it

How deep it goes
Works through stride sequences, access efficiency, and shared-memory bank mapping, including alignment and element-size qualifications.
Examples
Traces an eight-element reduction and scan, stride-2 warp reads, and particle field layouts.
What you can do after
Diagnose memory access spacing and evaluate layout changes while checking scan boundaries and synchronization.
How it connects
Connects stride to coalescing, bank conflicts, reduction, scan, and array-of-structures versus structure-of-arrays layouts.

Also explained in

Compare how 11 books explain stride →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.