liveBook · Manning

Element-wise multiplication

Element-wise multiplication multiplies corresponding elements of two compatible arrays or tensors independently.

How 2 Manning books explain it

GPU Programming with Triton

An elementwise product multiplies corresponding elements of compatible arrays.

How it is used
Uses corresponding-element multiplication to combine the activated gate and value tensors in SwiGLU.
How deep it goes
Defines the Hadamard product and applies it in SwiGLU, without developing general array compatibility or broadcasting rules.
Examples
Shows multiplying SiLU(G) by U and gives Python-style projection code for the SwiGLU output.
What you can do after
Build a fused SwiGLU elementwise stage in Triton using the activated gate and value inputs.

Read how GPU Programming with Triton explains element-wise multiplication →

Rearchitecting LLMs

Element-wise multiplication is the operation of multiplying corresponding entries of two equally sized vectors or arrays.

How it is used
Filters a content projection with a context-dependent gate inside a GLU MLP.
How deep it goes
Traces the gate and content projections through activation and multiplication to a same-sized, context-modulated vector.
Examples
Uses “spring” to show a gate emphasizing water and mountain meanings while attenuating other associations.
What you can do after
Understand how a gate modulates projected content through element-wise multiplication in a GLU MLP.

Read how Rearchitecting LLMs explains element-wise multiplication →

Also covered in

What to know first

Related concepts

Questions about element-wise multiplication

What is element-wise multiplication?

Element-wise multiplication multiplies corresponding elements of two compatible arrays or tensors independently.

Which Manning books explain element-wise multiplication?

GPU Programming with Triton and Rearchitecting LLMs.

What should I know before learning element-wise multiplication?

Manning books that teach element-wise multiplication usually introduce feedforward network, gradient, matrix-matrix multiplication and sigmoid first.

What is element-wise multiplication a kind of?

Element-wise multiplication is a kind of multiplication. Multiplication is a binary operation that combines two operands to produce their product.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.