GELU
GELU is a smooth neural-network activation function that scales an input by the standard normal cumulative distribution function evaluated at that input.
Rearchitecting LLMs
GELU is a smooth neural-network activation function that weights inputs using the Gaussian error function.
- How it is used
- Breaks linearity between expansion and contraction in a classic MLP by regulating neuron signals.
- How deep it goes
- Explains GELU’s filtering role and contrasts it with learned GLU gating, without supplying its formula.
- Examples
- Uses DistilGPT2’s 768-to-3072-to-768 MLP and contrasts fixed GELU filtering with GLU’s adaptive gate.
- What you can do after
- Understand why a classic MLP places GELU between expansion and contraction and how its filtering differs from learned gating.
Learn Generative AI with PyTorch
A smooth activation function used in neural networks to transform layer outputs and improve model performance.
- How it is used
- Uses a smooth nonlinear transformation inside Transformer feed-forward sublayers for language and music generation.
- How deep it goes
- Defines GELU, contrasts its differentiability with ReLU, and provides PyTorch code and a comparison plot without deriving the function.
- Examples
- Implements GELU in PyTorch, plots it against ReLU, and applies it in GPT-style text and music Transformers.
- What you can do after
- Builds a GELU module in PyTorch and places it within Transformer decoder feed-forward networks.
Build a Large Language Model (From Scratch)
A smooth activation function defined as x times the Gaussian CDF, used here in transformer feed-forward submodules.
- How it is used
- Activates hidden representations inside GPT-style transformer feed-forward networks.
- How deep it goes
- Defines the Gaussian-CDF form, implements a tanh approximation, compares ReLU behavior, and traces feed-forward tensor shapes.
- Examples
- Uses PyTorch code, GELU-versus-ReLU plots, and a [2, 3, 768] feed-forward shape example.
- What you can do after
- Build it from scratch as a PyTorch module and place it correctly in a GPT-style feed-forward network.
Read how Build a Large Language Model (From Scratch) explains GELU →
Also covered in
- Machines that ThinkIn early access (MEAP)
What to know first
- ReLUReLU is an activation function that maps an input x to the greater of x and zero.
- dropoutDropout is a neural-network regularization technique that randomly suppresses selected units or activations during training to reduce overfitting.
- dimensionA dimension is an independent coordinate or feature axis used to represent points or data; the dimensionality of a representation is the number of such axes or components.
- GPT modelA GPT model is a generative, autoregressive Transformer language model in the family developed by OpenAI.
Related concepts
- activation functionkind of · An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
- GPT-2used by · GPT-2 is a family of autoregressive, decoder-only Transformer language models developed by OpenAI that generate text by predicting successive tokens from preceding context.
- ReLUcontrasts with · ReLU is an activation function that maps an input x to the greater of x and zero.
- feedforward networkused by · A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.
Questions about GELU
What is GELU?
GELU is a smooth neural-network activation function that scales an input by the standard normal cumulative distribution function evaluated at that input.
Which Manning books explain GELU?
Rearchitecting LLMs, Learn Generative AI with PyTorch and Build a Large Language Model (From Scratch).
What should I know before learning GELU?
Manning books that teach GELU usually introduce ReLU, dropout, dimension and GPT model first.
What is the difference between GELU and GeGLU?
GELU is a smooth neural-network activation function that scales an input by the standard normal cumulative distribution function evaluated at that input. GeGLU is a gated activation function that multiplies a GELU-activated projection elementwise by a separate learned projection.
What is GELU a kind of?
GELU is a kind of activation function. An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
What is GELU also called?
GELU is also called Gaussian error linear unit.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


