liveBook · Manning

GELU

Also called Gaussian error linear unit

GELU is a smooth neural-network activation function that scales an input by the standard normal cumulative distribution function evaluated at that input.

How 3 Manning books explain it

Rearchitecting LLMs

GELU is a smooth neural-network activation function that weights inputs using the Gaussian error function.

How it is used
Breaks linearity between expansion and contraction in a classic MLP by regulating neuron signals.
How deep it goes
Explains GELU’s filtering role and contrasts it with learned GLU gating, without supplying its formula.
Examples
Uses DistilGPT2’s 768-to-3072-to-768 MLP and contrasts fixed GELU filtering with GLU’s adaptive gate.
What you can do after
Understand why a classic MLP places GELU between expansion and contraction and how its filtering differs from learned gating.

Read how Rearchitecting LLMs explains GELU →

Learn Generative AI with PyTorch

A smooth activation function used in neural networks to transform layer outputs and improve model performance.

How it is used
Uses a smooth nonlinear transformation inside Transformer feed-forward sublayers for language and music generation.
How deep it goes
Defines GELU, contrasts its differentiability with ReLU, and provides PyTorch code and a comparison plot without deriving the function.
Examples
Implements GELU in PyTorch, plots it against ReLU, and applies it in GPT-style text and music Transformers.
What you can do after
Builds a GELU module in PyTorch and places it within Transformer decoder feed-forward networks.

Read how Learn Generative AI with PyTorch explains GELU →

Build a Large Language Model (From Scratch)

A smooth activation function defined as x times the Gaussian CDF, used here in transformer feed-forward submodules.

How it is used
Activates hidden representations inside GPT-style transformer feed-forward networks.
How deep it goes
Defines the Gaussian-CDF form, implements a tanh approximation, compares ReLU behavior, and traces feed-forward tensor shapes.
Examples
Uses PyTorch code, GELU-versus-ReLU plots, and a [2, 3, 768] feed-forward shape example.
What you can do after
Build it from scratch as a PyTorch module and place it correctly in a GPT-style feed-forward network.

Read how Build a Large Language Model (From Scratch) explains GELU →

Also covered in

What to know first

Related concepts

Questions about GELU

What is GELU?

GELU is a smooth neural-network activation function that scales an input by the standard normal cumulative distribution function evaluated at that input.

Which Manning books explain GELU?

Rearchitecting LLMs, Learn Generative AI with PyTorch and Build a Large Language Model (From Scratch).

What should I know before learning GELU?

Manning books that teach GELU usually introduce ReLU, dropout, dimension and GPT model first.

What is the difference between GELU and GeGLU?

GELU is a smooth neural-network activation function that scales an input by the standard normal cumulative distribution function evaluated at that input. GeGLU is a gated activation function that multiplies a GELU-activated projection elementwise by a separate learned projection.

What is GELU a kind of?

GELU is a kind of activation function. An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.

What is GELU also called?

GELU is also called Gaussian error linear unit.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.