liveBook · Manning

Transposition

As explained in GPU Programming with Triton

A transpose exchanges a matrix’s rows and columns or, more generally, permutes a tensor’s dimensions.

A matrix transpose swaps its dimensions: for input A of shape M × N, output Aᵀ has shape N × M, with Aᵀ[j, i] = A[i, j]. The coordinate swap is simple; on a GPU, the challenge is keeping both reads and writes efficient. section 5.4 section 5.4.1

A basic Triton transpose kernel maps the operation onto tiles:

  • Assign each program instance a tile and form row and column coordinates from its program IDs. section 5.4.1
  • Use input strides to calculate load addresses and swap coordinates in the output address calculation. section 5.4.1
  • Mask coordinates beyond the matrix edges. For contiguous row-major input, neighboring lanes can read neighboring values, while the naive swapped-coordinate stores may jump between rows. section 5.4.1 section 5.4
Listing 5.3 Naive matrix transpose
import triton
import triton.language as tl

@triton.jit
def transpose_naive_kernel(x_ptr, y_ptr, M, N,
                           stride_xm, stride_xn, stride_ym, stride_yn,
                           BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr):
    pid_m = tl.program_id(axis=0)
    pid_n = tl.program_id(axis=1)

    rows = pid_m * BLOCK_M + tl.arange(0, BLOCK_M)[:, None]
    cols = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)[None, :]

    mask = (rows < M) & (cols < N)

    x_ptrs = x_ptr + rows * stride_xm + cols * stride_xn
    tile = tl.load(x_ptrs, mask=mask, other=0.0)

    y_ptrs = y_ptr + cols * stride_ym + rows * stride_yn
    tl.store(y_ptrs, tile, mask=mask)

The kernel makes the coordinate swap explicit; its shared tile layout for reads and writes leaves the naive store-access bottleneck. section 5.4.1

Tiling the reordering on chip can keep global-memory reads and writes coalesced. A transpose is useful when a later operation needs another dimension order, such as the QKᵀ product in attention. section 5.10 section 5.4

How GPU Programming with Triton covers it

How deep it goes
Defines the coordinate mapping and explains tiled, masked Triton indexing; details shared-memory bank handling remain brief.
Examples
Uses a 4×4 row-major access pattern, a Triton transpose kernel, and the QKᵀ attention product.
What you can do after
Build a basic Triton transpose kernel with stride-aware addresses, tiled coordinates, and boundary masks.
How it connects
Connects matrix transpose to strides, memory coalescing, shared memory tiling, and attention matrix products.

Also explained in

Compare how 9 books explain transposition →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.