Logits are the model’s unnormalized output scores for every vocabulary token. A GPT model produces them after embedding input tokens, adding positional information, processing transformer blocks, and applying final normalization and a linear output head. section 4.6
Follow the data flow from input text to generated text:
- A tokenizer converts text into token IDs. section 5.1.1
- The GPT model maps each input position to a vector of vocabulary-sized logits. section 4.6
- Generation converts logits into token IDs, then decodes those IDs into text. section 5.1.1
Shape in a GPT configuration
For a batch containing two sequences of four tokens, the model returns a tensor with shape [2, 4, 50257]. The final dimension represents one logit for each token in the 50,257-token vocabulary, while the first two dimensions preserve batch and sequence positions. section 4.6
Figure 4.15 An overview of the GPT model architecture showing the flow of data through the GPT model. Starting from the bottom, tokenized text is first converted into token embeddings, which are then augmented with positional embeddings. This combined information forms a tensor that is passed through a series of transformer blocks shown in the center (each containing multi-head attention and feed forward neural network layers with dropout and layer normalization), which are stacked on top of each other and repeated 12 times.
Trace how token embeddings and positional embeddings move through stacked transformer blocks before producing vocabulary-level logits. section 4.6
