GELU is a smooth, nonlinear activation used between expansion and contraction in a classic MLP. By regulating which neuron signals pass through and at what intensity, it breaks the linearity that would otherwise let the two transformations collapse into one matrix multiplication. section 3.1.3
DistilGPT2 MLP
The MLP expands from 768 to 3072 dimensions, applies GELU, then contracts back to 768. In contrast, GELU applies fixed filtering criteria regardless of context; GLU adds a learned gate that adapts filtering to the input. section 3.1.3 section 3.2.2
(mlp): GPT2MLP(
(c_fc): Conv1D(nf=3072, nx=768)
(c_proj): Conv1D(nf=768, nx=3072)
(act): NewGELUActivation()
(dropout): Dropout(p=0.1, inplace=False)
)
Inspect the classic MLP’s layer dimensions and NewGELUActivation placement. section 3.1.3


