A matrix transpose swaps its dimensions: for input A of shape M × N, output Aᵀ has shape N × M, with Aᵀ[j, i] = A[i, j]. The coordinate swap is simple; on a GPU, the challenge is keeping both reads and writes efficient. section 5.4 section 5.4.1
A basic Triton transpose kernel maps the operation onto tiles:
- Assign each program instance a tile and form row and column coordinates from its program IDs. section 5.4.1
- Use input strides to calculate load addresses and swap coordinates in the output address calculation. section 5.4.1
- Mask coordinates beyond the matrix edges. For contiguous row-major input, neighboring lanes can read neighboring values, while the naive swapped-coordinate stores may jump between rows. section 5.4.1 section 5.4
Listing 5.3 Naive matrix transpose
import triton
import triton.language as tl
@triton.jit
def transpose_naive_kernel(x_ptr, y_ptr, M, N,
stride_xm, stride_xn, stride_ym, stride_yn,
BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr):
pid_m = tl.program_id(axis=0)
pid_n = tl.program_id(axis=1)
rows = pid_m * BLOCK_M + tl.arange(0, BLOCK_M)[:, None]
cols = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)[None, :]
mask = (rows < M) & (cols < N)
x_ptrs = x_ptr + rows * stride_xm + cols * stride_xn
tile = tl.load(x_ptrs, mask=mask, other=0.0)
y_ptrs = y_ptr + cols * stride_ym + rows * stride_yn
tl.store(y_ptrs, tile, mask=mask)
The kernel makes the coordinate swap explicit; its shared tile layout for reads and writes leaves the naive store-access bottleneck. section 5.4.1
Tiling the reordering on chip can keep global-memory reads and writes coalesced. A transpose is useful when a later operation needs another dimension order, such as the QKᵀ product in attention. section 5.10 section 5.4








