Element-wise multiplication
Element-wise multiplication multiplies corresponding elements of two compatible arrays or tensors independently.
GPU Programming with Triton
An elementwise product multiplies corresponding elements of compatible arrays.
- How it is used
- Uses corresponding-element multiplication to combine the activated gate and value tensors in SwiGLU.
- How deep it goes
- Defines the Hadamard product and applies it in SwiGLU, without developing general array compatibility or broadcasting rules.
- Examples
- Shows multiplying SiLU(G) by U and gives Python-style projection code for the SwiGLU output.
- What you can do after
- Build a fused SwiGLU elementwise stage in Triton using the activated gate and value inputs.
Read how GPU Programming with Triton explains element-wise multiplication →
Rearchitecting LLMs
Element-wise multiplication is the operation of multiplying corresponding entries of two equally sized vectors or arrays.
- How it is used
- Filters a content projection with a context-dependent gate inside a GLU MLP.
- How deep it goes
- Traces the gate and content projections through activation and multiplication to a same-sized, context-modulated vector.
- Examples
- Uses “spring” to show a gate emphasizing water and mountain meanings while attenuating other associations.
- What you can do after
- Understand how a gate modulates projected content through element-wise multiplication in a GLU MLP.
Read how Rearchitecting LLMs explains element-wise multiplication →
Also covered in
What to know first
- feedforward networkA feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- sigmoidThe logistic sigmoid is a smooth, S-shaped function that maps real-valued inputs to values between zero and one.
Related concepts
- link predictionuses · Link prediction is a graph-learning task that estimates whether a connection exists or may occur between nodes, often by scoring candidate links.
- tensor operationkind of · A tensor operation is a mathematical or computational transformation applied to one or more tensors, producing tensor or scalar values.
Questions about element-wise multiplication
What is element-wise multiplication?
Element-wise multiplication multiplies corresponding elements of two compatible arrays or tensors independently.
Which Manning books explain element-wise multiplication?
GPU Programming with Triton and Rearchitecting LLMs.
What should I know before learning element-wise multiplication?
Manning books that teach element-wise multiplication usually introduce feedforward network, gradient, matrix-matrix multiplication and sigmoid first.
What is element-wise multiplication a kind of?
Element-wise multiplication is a kind of multiplication. Multiplication is a binary operation that combines two operands to produce their product.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

