liveBook · Manning

Vector addition

As explained in GPU Programming with Triton

Vector addition is the element-wise operation that adds corresponding components of two equal-sized vectors to produce a third vector.

Vector addition adds corresponding elements of two equal-sized vectors: for inputs X and Y of size N, each output element is zᵢ = xᵢ + yᵢ. section 2.2

\[\begin{aligned} X &= [x_1, x_2, x_3, \ldots, x_n], \quad Y = [y_1, y_2, y_3, \ldots, y_n] \\ Z &= X + Y = [z_1, z_2, z_3, \ldots, z_n] \\ z_1 &= x_1 + y_1, \\ z_2 &= x_2 + y_2, \\ z_3 &= x_3 + y_3, \\ &\vdots \\ z_n &= x_n + y_n. \end{aligned}\]

The element-wise definition makes clear that each output depends on the matching elements of the two inputs. section 2.2

On a GPU, Triton divides the elements into contiguous blocks handled by independent program instances. Each instance calculates offsets, loads its input slices, adds them, and stores the matching output slice; masks handle elements beyond the vector length. section 2.4

The operation is a clear way to learn kernel construction, but a library operation is preferable for basic addition. Triton’s advantage is combining multiple steps in one kernel to reduce memory movement and launches. section 2.2

Benchmarking implementations

For 98,432-element vectors, compare Python, PyTorch, and Triton. Check the Triton result against PyTorch, then benchmark with GPU synchronization or Triton’s benchmarking utility so asynchronous launches and first-call compilation do not distort timing. section 2.2.10

How GPU Programming with Triton covers it

How deep it goes
Explains the element-wise definition and block-level kernel mapping, then defers fused workloads to later treatment.
Examples
Compares Python, PyTorch, and Triton implementations on vectors, including a 98,432-element correctness and timing example.
What it assumes
Assumes intermediate Python knowledge and prior grounding in GPU organization.
What you can do after
Build it from scratch as a Triton kernel and host wrapper, then check and benchmark its output.
How it connects
Connects vector addition to Triton kernels, the SPMD model, PyTorch, and GPU parallel execution.

Also explained in

Compare how 2 books explain vector addition →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.