GELU, or Gaussian error linear unit, is a smooth activation function defined as GELU(x) = x·Φ(x), where Φ is the standard Gaussian cumulative distribution function. Practical implementations commonly use a cheaper approximation. section 4.3
In a GPT transformer block, GELU sits between two linear layers in the feed-forward network. The first expands each token representation, the activation transforms it smoothly, and the second projects it back to the embedding size. section 4.3 section 4.8
Compared with ReLU, GELU:
- Provides a smooth nonlinear response rather than a piecewise-linear cutoff. section 4.3
- Maintains a non-zero gradient across almost all negative inputs, supporting more nuanced parameter updates. section 4.3
Feed-forward shape flow
With an input shaped [2, 3, 768], a GPT-style feed-forward module expands the final dimension to 3,072, applies GELU, and contracts it back to [2, 3, 768]. section 4.3
Listing 4.3 An implementation of the GELU activation function
class GELU(nn.Module):
def __init__(self):
super().__init__()
def forward(self, x):
return 0.5 * x * (1 + torch.tanh(
torch.sqrt(torch.tensor(2.0 / torch.pi)) *
(x + 0.044715 * torch.pow(x, 3))
))
Shows a PyTorch GELU module using the computationally cheaper tanh-based approximation. section 4.3
