liveBook · Manning

Element-wise multiplication

As explained in Rearchitecting LLMs

Element-wise multiplication is the operation of multiplying corresponding entries of two equally sized vectors or arrays.

Element-wise multiplication combines corresponding entries of equally sized vectors. In a GLU MLP, it combines a content projection with a SiLU-activated gate, producing a same-sized vector whose content is modulated by context. section 3.2.2

Contextual filtering

For the token “spring” in “Fresh water flows from the mountain spring daily,” the gate can emphasize water- and mountain-related content while attenuating other meanings. section 3.2.2

Figure 3.8 Processing flow in MLP Modules: classic vs. GLU. While the classic MLP follows a simple expansion-activation-contraction sequence, GLU uses two parallel projections: one up_proj that transforms the content and another gate_proj that creates a dynamic filter.

Compare the classic MLP flow with the GLU’s parallel content and gate projections. section 3.2.2

How Rearchitecting LLMs covers it

How deep it goes
Traces the gate and content projections through activation and multiplication to a same-sized, context-modulated vector.
Examples
Uses “spring” to show a gate emphasizing water and mountain meanings while attenuating other associations.
What you can do after
Understand how a gate modulates projected content through element-wise multiplication in a GLU MLP.
How it connects
Connects GLU and MLP layers with SiLU activation and content and gate projections.

Also explained in

Compare how 2 books explain element-wise multiplication →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.