liveBook · Manning

GELU

As explained in Rearchitecting LLMs

GELU is a smooth neural-network activation function that weights inputs using the Gaussian error function.

GELU is a smooth, nonlinear activation used between expansion and contraction in a classic MLP. By regulating which neuron signals pass through and at what intensity, it breaks the linearity that would otherwise let the two transformations collapse into one matrix multiplication. section 3.1.3

DistilGPT2 MLP

The MLP expands from 768 to 3072 dimensions, applies GELU, then contracts back to 768. In contrast, GELU applies fixed filtering criteria regardless of context; GLU adds a learned gate that adapts filtering to the input. section 3.1.3 section 3.2.2

(mlp): GPT2MLP(
     (c_fc): Conv1D(nf=3072, nx=768)
     (c_proj): Conv1D(nf=768, nx=3072)
     (act): NewGELUActivation()
     (dropout): Dropout(p=0.1, inplace=False)
)

Inspect the classic MLP’s layer dimensions and NewGELUActivation placement. section 3.1.3

How Rearchitecting LLMs covers it

How deep it goes
Explains GELU’s filtering role and contrasts it with learned GLU gating, without supplying its formula.
Examples
Uses DistilGPT2’s 768-to-3072-to-768 MLP and contrasts fixed GELU filtering with GLU’s adaptive gate.
What you can do after
Understand why a classic MLP places GELU between expansion and contraction and how its filtering differs from learned gating.
How it connects
Connects GELU to classic MLP expansion and contraction, and contrasts it with GLU and its variants.

Also explained in

Compare how 3 books explain GELU →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.