Model architecture is the internal organization of a neural model: the arrangement that determines how inputs are processed and outputs are produced. Understanding that organization helps replace black-box use with practical insight into how generative models work. section 1.4
A useful way to study architecture is to build the model from scratch. This exposes its design choices and makes the model easier to understand and use responsibly than treating it as a black box. section 1.4
A GPT architecture can be connected directly to its training and generation workflow:
- Represent training text as integer-token sequences of a fixed length. section 12.1
- Shift each input sequence one token to the right to create the target outputs. section 12.1
- Train the model to predict the next token from the current token and all previous tokens. section 12.1
- Generate text by converting a prompt into token indexes, predicting tokens iteratively, and converting the result back into text. section 12.1
GPT text generation
Given a text prompt, the trained GPT model repeatedly predicts the most likely next token. The resulting token sequence is then converted into readable text. section 12.1
