liveBook · Manning

Tiled transpose

As explained in GPU Programming with Triton

A tiled transpose rearranges a matrix tile so its elements are written in transposed order while preserving efficient memory access.

A transpose can compute correct output addresses yet perform poorly if neighboring GPU lanes access strided global-memory locations. A tiled transpose reorders values on chip so both the input load and output store can access neighboring addresses together. section 5.10 section 5.4.3

In a low-level shared-memory implementation, threads load a tile in coalesced order, synchronize, then read it transposed and store it in coalesced order. Synchronization ensures every tile value is written and visible before threads begin the transposed read. Triton expresses the operation with whole-tile loads and stores: `tl.make_block_ptr` describes each input and output block, `tl.trans` reorders the loaded tile, and `boundary_check` handles edge tiles. The Triton kernel does not declare a shared-memory array. section 5.4.3 section 5.4.3

Transpose one tile

For a tile at program IDs `(pid_m, pid_n)`, load input offsets `(pid_m * BLOCK_M, pid_n * BLOCK_N)` with block shape `(BLOCK_M, BLOCK_N)`. Store `tl.trans(tile)` at output offsets `(pid_n * BLOCK_N, pid_m * BLOCK_M)` with the block dimensions reversed. Boundary checks protect incomplete tiles. section 5.4.3

Figure 5.9 Shared-memory staging for a tiled transpose. The kernel loads an input row contiguously, changes the traversal direction on chip, and writes the corresponding output row contiguously. The highlighted values show how input column 0 becomes output row 0.
CH05 F09 Fartale

Trace how a contiguous input row is reordered on chip before a contiguous output row is written. section 5.4.3

How GPU Programming with Triton covers it

How deep it goes
Explains the coalescing problem, shared-memory synchronization, and a Triton block-pointer kernel with edge checks.
Examples
Uses shared-memory pseudocode, a Triton kernel, and a diagram tracing a tile from input to output.
What you can do after
Build it from scratch by loading a tile, transposing it, and storing it with reversed offsets and boundary checks.
How it connects
Connects memory coalescing with shared memory, block pointers, and tile transposition.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.