A computation is embarrassingly parallel when it splits into independent tasks that need no communication, synchronization, or shared state. For an elementwise operation, each output depends only on its corresponding input, so many workers can apply the same function to different elements. section 1.3.2 section 3.1.1
Common elementwise examples include:
- ReLU, sigmoid, GELU, and arithmetic on paired elements. section 3.1.1
- SiLU, where each input element is transformed independently. section 3.1.2
SiLU in Triton
A SiLU kernel loads a block, computes the activation in registers, and stores the results without communication between program instances. Masked loads and stores handle a final partial block; ceiling division launches enough instances to cover the input. section 3.1.2
Listing 3.1 SiLU kernel in Triton
import torch
import triton
import triton.language as tl
@triton.jit
def silu_kernel(input_ptr, output_ptr,
n_elements,
BLOCK_SIZE: tl.constexpr): #1
pid = tl.program_id(0) #2
block_start = pid * BLOCK_SIZE
offsets = block_start + tl.arange(0,
BLOCK_SIZE) #3
mask = offsets < n_elements
x = tl.load(input_ptr + offsets,
mask=mask) #4
y = tl.sigmoid(x) * x #5
tl.store(output_ptr + offsets,
y,
mask=mask) #6
def solve(input_tensor: torch.Tensor,
output_tensor: torch.Tensor,
N: int):
BLOCK_SIZE = 1024
grid = (triton.cdiv(N, BLOCK_SIZE),) #7
silu_kernel[grid](input_tensor, output_tensor, N,
BLOCK_SIZE) #8
Inspect the complete Triton kernel and its launch wrapper. section 3.1.2


