Parameter count measures a model’s learned numerical parameters. It helps compare model scale and, for dense transformer feed-forward networks, relates to the computation used for each input token. section 4.1.1 section 8.6.1
Parameter totals help make architectural trade-offs concrete:
- Dense FFNs activate all their parameters for each input token, contributing to training cost and inference latency. section 4.1.1
- In a toy distillation setup, the teacher has roughly 26,000 parameters and the student roughly 8,800; the student is about three times smaller. section 8.6.1
Figure 4.2 The standard Feed-Forward Network (FFN) in a Transformer block, featuring an expansion-contraction architecture.

Shows the expansion-contraction structure of a dense transformer FFN, the architecture whose per-token parameter use motivates the cost comparison. section 4.1.1
Figure 8.21 The toy-sized teacher model (4 layers, 128-dim hidden, ~26K parameters) and student model (2 layers, 32-dim hidden, ~8.8K parameters) used in our implementation. The student is 3x smaller but maintains the same input and output dimensions.

Compares the toy teacher and student architectures alongside their approximate parameter totals. section 8.6.1





