A Transformer model has an input, a stack of Transformer blocks, and an output. Attention handles context and creates an inference cache that grows with each token; the MLP processes knowledge and consumes most model parameters and compute. section 3.0 section 3.1
A static model structure maps to these main parts:
- Input embeddings represent tokens and positions; repeated Transformer blocks contain attention and MLP components. section 3.1
- The language-modeling head belongs to the wrapper above the GPT2Model base model. section 3.1
- Weights are learned static parameters; activations are input-dependent values passed through the network. section 3.3
Figure 3.9 How activations flow between layers. Each neuron applies its static weights (learned during training) to the input values and passes the result through an activation function (SiLU). The output values, the activations, vary in intensity: high values propagate strong signals, while low values suppress information. These activations become the inputs for the next layer, creating the continuous flow of information through the model.
Trace how layer outputs become activations that feed the next layer. section 3.3







































