A feedforward network transforms inputs through learned layers without recurrent state. Its common multilayer-perceptron form combines linear operations and activations to detect increasingly complex features. section 2.2.6 section 9.1.3
Practical characteristics include:
- Stacking fully connected layers increases representational capacity, but adding parameters alone does not guarantee better results. section 2.2.6
- Compared with Llama-style models, the simple version omits or simplifies normalization, attention, activation choices, and layer count. section 9.5
Small text generator
A PyTorch model can map embedded tokens through a two-layer feedforward stack and train with cross-entropy while ignoring padding tokens. A short training run produces poorly formed text, illustrating that a working architecture is not automatically a useful language model.
Listing 9.3 Simple model and training loop
class SimpleFeedForwardNN(nn.Module):
def __init__(self, config=MASTER_CONFIG):
super().__init__()
self.config = config
self.embedding = nn.Embedding(
config["vocab_size"], config["d_model"]
)
self.linear = nn.Sequential(
nn.Linear(config["d_model"], config["d_model"]),
nn.ReLU(),
nn.Linear(config["d_model"], config["vocab_size"]),
)
print(
f"model params: {sum([m.numel() for m in self.parameters()])}"
)
def forward(self, idx, targets=None):
x = self.embedding(idx)
logits = self.linear(x)
if targets is not None:
loss = F.cross_entropy(
logits.view(-1, self.config["vocab_size"]),
targets.view(-1),
ignore_index=tokenizer.pad_token_id,
# reduction="sum",
)
return logits, loss
else:
return logits
model = SimpleFeedForwardNN(MASTER_CONFIG).to(device)
opt_model = torch.compile(model) #1
Inspect the compact PyTorch implementation showing embeddings, linear layers, ReLU, vocabulary logits, and padding-aware loss. section 9.1.3
