liveBook · Manning

Element-wise operation

As explained in GPU Programming with Triton

An element-wise tensor operation applies the same function independently to individual tensor elements.

An element-wise operation applies the same function independently at each tensor index: output[i] = f(input[i]). Each output depends only on the corresponding input, so elements need no coordination or synchronization. section 3.1.1

This independence makes element-wise work a baseline GPU pattern: threads perform the same operation on different data. The workload maps cleanly to SIMT execution and can scale across many cores, but is often limited by memory bandwidth. section 1.3.2 section 3.1.1

Vector addition formula
\[ X = [x_1, x_2, x_3, \ldots, x_n], \quad Y = [y_1, y_2, y_3, \ldots, y_n] \] \[ Z = X + Y = [z_1, z_2, z_3, \ldots, z_n] \] \[ \begin{aligned} and\vdots \\ z_n and= x_n + y_n. \end{aligned} \]

Vector addition makes the index-by-index correspondence explicit: each output element combines inputs at the same index. section 3.1.1

SiLU and element-wise fusion

SiLU applies x · sigmoid(x) independently to each element. A Triton kernel can load a block, compute SiLU, and store results; a mask handles the final partial block. Chaining another element-wise operation in a separate kernel can add another global-memory round trip, motivating fusion. section 3.1.2 section 3.1.2

Listing 3.1 SiLU kernel in Triton
import torch
import triton
import triton.language as tl

@triton.jit
def silu_kernel(input_ptr, output_ptr,
                n_elements,
                BLOCK_SIZE: tl.constexpr): #1
    pid = tl.program_id(0) #2
    block_start = pid * BLOCK_SIZE
    offsets = block_start + tl.arange(0,
                                     BLOCK_SIZE) #3
    mask = offsets < n_elements

    x = tl.load(input_ptr + offsets,
                mask=mask) #4
    y = tl.sigmoid(x) * x #5
    tl.store(output_ptr + offsets,
             y,
             mask=mask) #6

def solve(input_tensor: torch.Tensor,
          output_tensor: torch.Tensor,
          N: int):
    BLOCK_SIZE = 1024
    grid = (triton.cdiv(N, BLOCK_SIZE),) #7
    silu_kernel[grid](input_tensor, output_tensor, N,
                      BLOCK_SIZE) #8

The Triton listing shows the load-compute-store pattern, block indexing, and masking for a concrete element-wise kernel. section 3.1.2

How GPU Programming with Triton covers it

How deep it goes
Defines index independence and explains SIMT mapping, then demonstrates a masked Triton kernel without detailing hardware internals.
Examples
Uses vector addition, ReLU, sigmoid, GELU, and a SiLU kernel as concrete element-wise examples.
What you can do after
Understand how independent tensor elements map to GPU work and why chaining operations can increase memory traffic.
How it connects
Connects element-wise operations to embarrassingly parallel tasks, SIMT execution, memory bandwidth, and kernel fusion.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.