How deeply does LLMs in Production cover SwiGLU?

Explains its activation role and placement, but defers the internal implementation to a notebook.

SwiGLU

SWIGLU is a gated activation function that uses the Swish function to modulate one linear projection with another.

SwiGLU is a gated activation that uses Swish to modulate one linear projection with another. In a Llama-style network, it replaces ReLU, supports negative values, and helps address exploding or vanishing gradients. section 9.2

SwiGLU in a Llama block

A SimpleLlama feed-forward path projects from d_model to hidden_dim, applies SwiGLU, then projects back to d_model. The same activation appears in the model’s final feed-forward network instead of ReLU.

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

ReLUalternative to · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
matrix-matrix multiplicationuses · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
exploding gradientaddresses · An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or numerical and convergence failures.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.