GELU
GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
Learn Generative AI with PyTorch
A smooth activation function used in neural networks to transform layer outputs and improve model performance.
- How it is used
- Uses a smooth nonlinear transformation inside Transformer feed-forward sublayers for language and music generation.
- How deep it goes
- Defines GELU, contrasts its differentiability with ReLU, and provides PyTorch code and a comparison plot without deriving the function.
- Examples
- Implements GELU in PyTorch, plots it against ReLU, and applies it in GPT-style text and music Transformers.
- What you can do after
- Builds a GELU module in PyTorch and places it within Transformer decoder feed-forward networks.
Build a Large Language Model (From Scratch)
A smooth activation function defined as x times the Gaussian CDF, used here in transformer feed-forward submodules.
- How it is used
- Activates hidden representations inside GPT-style transformer feed-forward networks.
- How deep it goes
- Defines the Gaussian-CDF form, implements a tanh approximation, compares ReLU behavior, and traces feed-forward tensor shapes.
- Examples
- Uses PyTorch code, GELU-versus-ReLU plots, and a [2, 3, 768] feed-forward shape example.
- What you can do after
- Build it from scratch as a PyTorch module and place it correctly in a GPT-style feed-forward network.
Read how Build a Large Language Model (From Scratch) explains GELU →
Related concepts
- activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
- GPT-2used by · GPT-2 is a family of autoregressive, decoder-only Transformer language models developed by OpenAI that generate text by predicting successive tokens from preceding context.
- ReLUcontrasts with · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
- feedforward networkused by · A feedforward network is a neural network or subnetwork in which information flows from inputs toward outputs through successive computations without recurrent connections or cycles.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.