GELU (Gaussian error linear unit) is a smooth activation function used in Transformer feed-forward networks. It combines linear and nonlinear behavior to make more nuanced adjustments during training and can improve performance, especially on language tasks. section 11.2 section 11.2.2
Compared with ReLU, GELU is differentiable everywhere rather than having a kink. Its smoother gradient provides a more continuous signal for backpropagation and may optimize the network more effectively. section 11.2.2
Implementing GELU in PyTorch
Define GELU as an nn.Module and apply its tanh-based approximation to an input tensor x. Plotting its output beside ReLU makes the difference between the smooth curve and ReLU's kink visible. section 11.2.2
class GELU(nn.Module):
def forward(self, x):
return 0.5*x*(1.0+torch.tanh(math.sqrt(2.0/math.pi)*\
(x + 0.044715 * torch.pow(x, 3.0))))
Inspect the compact PyTorch implementation, including the cubic term, scaling factor, and tensor-valued forward computation. section 11.2.2
Use GELU in the feed-forward sublayer of decoder blocks for GPT-style text models and music Transformers. The same activation pattern appears across models with different layer counts, embedding sizes, vocabularies, and sequence lengths. section 12.3.1 section 14.3.1
