A GPT model is a decoder-only generative pretrained transformer that produces text one token at a time. It processes context left to right, using previous tokens to predict the next one. section 1.4 section 1.6
The model uses token and positional embeddings, repeated transformer blocks with masked multi-head attention, final normalization, and an output projection over the vocabulary. Next-token pretraining creates labels directly from unlabeled text. section 1.6 section 4.1
A GPT implementation follows this data flow:
- Tokenize input sequences and represent each token with an embedding. section 4.1
- Add positional embeddings and apply dropout to preserve sequence-position information. section 4.1
- Pass the sequence through repeated transformer blocks using masked attention and normalization. section 4.0 section 4.1
- Project each contextual representation to vocabulary logits for next-token prediction. section 4.1
A small GPT-shaped forward pass
With two four-token sequences and a 50,257-token vocabulary, a placeholder GPT model produces logits shaped [2, 4, 50257]: one vocabulary score vector for every token position in every sequence. section 4.1
Figure 4.2 A GPT model. In addition to the embedding layers, it consists of one or more transformer blocks containing the masked multi-head attention module we previously implemented.
Use the architecture diagram to trace embeddings into repeated transformer blocks containing masked multi-head attention. section 4.1
