liveBook · Manning

Embarrassingly parallel

As explained in GPU Programming with Triton

An embarrassingly parallel task is a computation whose output elements depend independently on separate inputs and require no coordination between work instances.

A computation is embarrassingly parallel when it splits into independent tasks that need no communication, synchronization, or shared state. For an elementwise operation, each output depends only on its corresponding input, so many workers can apply the same function to different elements. section 1.3.2 section 3.1.1

Common elementwise examples include:

  • ReLU, sigmoid, GELU, and arithmetic on paired elements. section 3.1.1
  • SiLU, where each input element is transformed independently. section 3.1.2

SiLU in Triton

A SiLU kernel loads a block, computes the activation in registers, and stores the results without communication between program instances. Masked loads and stores handle a final partial block; ceiling division launches enough instances to cover the input. section 3.1.2

Listing 3.1 SiLU kernel in Triton
import torch
import triton
import triton.language as tl

@triton.jit
def silu_kernel(input_ptr, output_ptr,
                n_elements,
                BLOCK_SIZE: tl.constexpr): #1
    pid = tl.program_id(0) #2
    block_start = pid * BLOCK_SIZE
    offsets = block_start + tl.arange(0,
                                     BLOCK_SIZE) #3
    mask = offsets < n_elements

    x = tl.load(input_ptr + offsets,
                mask=mask) #4
    y = tl.sigmoid(x) * x #5
    tl.store(output_ptr + offsets,
             y,
             mask=mask) #6

def solve(input_tensor: torch.Tensor,
          output_tensor: torch.Tensor,
          N: int):
    BLOCK_SIZE = 1024
    grid = (triton.cdiv(N, BLOCK_SIZE),) #7
    silu_kernel[grid](input_tensor, output_tensor, N,
                      BLOCK_SIZE) #8

Inspect the complete Triton kernel and its launch wrapper. section 3.1.2

How GPU Programming with Triton covers it

How deep it goes
Defines independence, maps it to SIMT, and walks through a SiLU kernel without detailing hardware scheduling.
Examples
Uses ReLU, sigmoid, GELU, paired-element arithmetic, and a Triton SiLU kernel as examples.
What you can do after
Build from scratch a Triton SiLU kernel with masked block accesses and a ceiling-divided launch grid.
How it connects
Connects data parallelism, SIMT execution, elementwise operations, and kernel fusion.

Also explained in

Compare how 3 books explain embarrassingly parallel →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.