An element-wise operation applies the same function independently at each tensor index: output[i] = f(input[i]). Each output depends only on the corresponding input, so elements need no coordination or synchronization. section 3.1.1
This independence makes element-wise work a baseline GPU pattern: threads perform the same operation on different data. The workload maps cleanly to SIMT execution and can scale across many cores, but is often limited by memory bandwidth. section 1.3.2 section 3.1.1
Vector addition formula
Vector addition makes the index-by-index correspondence explicit: each output element combines inputs at the same index. section 3.1.1
SiLU and element-wise fusion
SiLU applies x · sigmoid(x) independently to each element. A Triton kernel can load a block, compute SiLU, and store results; a mask handles the final partial block. Chaining another element-wise operation in a separate kernel can add another global-memory round trip, motivating fusion. section 3.1.2 section 3.1.2
Listing 3.1 SiLU kernel in Triton
import torch
import triton
import triton.language as tl
@triton.jit
def silu_kernel(input_ptr, output_ptr,
n_elements,
BLOCK_SIZE: tl.constexpr): #1
pid = tl.program_id(0) #2
block_start = pid * BLOCK_SIZE
offsets = block_start + tl.arange(0,
BLOCK_SIZE) #3
mask = offsets < n_elements
x = tl.load(input_ptr + offsets,
mask=mask) #4
y = tl.sigmoid(x) * x #5
tl.store(output_ptr + offsets,
y,
mask=mask) #6
def solve(input_tensor: torch.Tensor,
output_tensor: torch.Tensor,
N: int):
BLOCK_SIZE = 1024
grid = (triton.cdiv(N, BLOCK_SIZE),) #7
silu_kernel[grid](input_tensor, output_tensor, N,
BLOCK_SIZE) #8
The Triton listing shows the load-compute-store pattern, block indexing, and masking for a concrete element-wise kernel. section 3.1.2
