ReLU, or rectified linear unit, outputs zero for negative inputs and otherwise passes the input through. It commonly sits inside a neural network as an activation function. section 2.2.4 section 9.1.3
ReLU is a practical default, not a universal answer: it helps with vanishing gradients but leaves exploding gradients unresolved and removes negative comparisons from the model. section 2.2.4
Choose ReLU with these constraints in mind:
- Use it when a simple, familiar activation is appropriate for a neural-network training loop. section 2.2.4
- Do not treat it as a solution to exploding gradients; that failure mode remains. section 2.2.4
- Consider ELU when retaining negative values matters, or GEGLU and SWIGLU for increasingly difficult language scenarios. section 2.2.4
Two small language models
A NumPy continuous CBoW model places ReLU alongside weight initialization, softmax, forward propagation, and backpropagation. A compact feed-forward language model places ReLU between two linear layers before producing vocabulary logits. section 2.2.4 section 9.1.3
Listing 9.3 Simple model and training loop
class SimpleFeedForwardNN(nn.Module):
def __init__(self, config=MASTER_CONFIG):
super().__init__()
self.config = config
self.embedding = nn.Embedding(
config["vocab_size"], config["d_model"]
)
self.linear = nn.Sequential(
nn.Linear(config["d_model"], config["d_model"]),
nn.ReLU(),
nn.Linear(config["d_model"], config["vocab_size"]),
)
print(
f"model params: {sum([m.numel() for m in self.parameters()])}"
)
def forward(self, idx, targets=None):
x = self.embedding(idx)
logits = self.linear(x)
if targets is not None:
loss = F.cross_entropy(
logits.view(-1, self.config["vocab_size"]),
targets.view(-1),
ignore_index=tokenizer.pad_token_id,
# reduction="sum",
)
return logits, loss
else:
return logits
model = SimpleFeedForwardNN(MASTER_CONFIG).to(device)
opt_model = torch.compile(model) #1
Inspect the simple model implementation to see ReLU positioned between the embedding projection and vocabulary-output layer, alongside cross-entropy loss. section 9.1.3
