liveBook · Manning

Element-wise multiplication

As explained in GPU Programming with Triton

An elementwise product multiplies corresponding elements of compatible arrays.

Element-wise multiplication pairs corresponding positions in two arrays and multiplies each pair. In SwiGLU, the operation combines the SiLU-activated gate tensor with the value tensor: output = SiLU(G) ⊙ U. Fusing this step with its surrounding computation can avoid materializing intermediate tensors in global memory. section 3.3.1 section 3.3.1

Simplified SwiGLU
\[\mathrm{SwiGLU}(x) = \underbrace{\mathrm{SiLU}(G)}_{\text{Gate}} \odot \underbrace{U}_{\text{Value}}\]

The simplified formula shows the element-wise product between the activated gate and the value. section 3.3.1

SwiGLU in Python

gate = silu(x @ W); up = x @ V; output = gate * up. The multiplication combines matching elements of the two projection outputs. section 3.3.1

How GPU Programming with Triton covers it

How deep it goes
Defines the Hadamard product and applies it in SwiGLU, without developing general array compatibility or broadcasting rules.
Examples
Shows multiplying SiLU(G) by U and gives Python-style projection code for the SwiGLU output.
What it assumes
Takes gate and up-projection tensors produced by preceding matrix multiplications as inputs to the elementwise stage.
What you can do after
Build a fused SwiGLU elementwise stage in Triton using the activated gate and value inputs.
How it connects
Connects element-wise multiplication to SwiGLU, SiLU, Swish, and preceding matrix multiplications.

Also explained in

Compare how 2 books explain element-wise multiplication →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.