liveBook · Manning

Broadcast operation

As explained in GPU Programming with Triton

Broadcasting makes tensor values behave as though they were repeated across compatible dimensions without necessarily copying the underlying data.

Broadcasting lets values with compatible shapes participate in a larger tensor operation without requiring repeated copies. In a PyTorch view, a stride of zero means an index can advance along a logical dimension without changing the memory address. For example, adding a vector of shape (N,) to a matrix of shape (M, N) treats the vector as a view of shape (M, N) with strides (0, 1): every row reuses the same stored vector. section 5.2.3

In Triton, broadcasting also builds multidimensional index and pointer tensors. Given row offsets of shape (BLOCK_M,) and column offsets of shape (BLOCK_N,), add a dimension to each to obtain shapes (BLOCK_M, 1) and (1, BLOCK_N). Broadcasting their stride-scaled values produces a (BLOCK_M, BLOCK_N) array of offsets, with one address for each tile element. section 5.3.2

Figure 5.6 Constructing the 2D pointer offset table by broadcasting. row_offs × stride_m has shape BLOCK_M × 1, and col_offs × stride_n has shape 1 × BLOCK_N. Broadcasting adds the two contributions at every position in the BLOCK_M × BLOCK_N offset table. For a row-major tile at the origin with stride_m = 4 and stride_n = 1, element (1, 2) receives offset 4 + 2 = 6.
CH05 F06 Fartale

The offset table shows how row and column contributions broadcast to form a two-dimensional pointer array. section 5.3.2

How GPU Programming with Triton covers it

How deep it goes
Works through zero-stride views and expanded offset vectors, stopping at pointer construction for a tiled load.
Examples
Uses vector-to-matrix addition and row-plus-column offsets to build a two-dimensional tile pointer array.
What you can do after
Apply it via Triton’s expand_dims or None indexing to construct broadcasted tile pointers.
How it connects
Connects tensor strides and PyTorch views to Triton indexing, pointer arithmetic, and softmax reduction-broadcast patterns.

Also explained in

Compare how 5 books explain broadcast operation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.