Expressive power is an architecture’s ability to represent diverse functions, patterns, or input-output relationships. In multi-head attention, independent key and value projections let heads specialize in different relationships and blend them into richer context vectors. section 2.6.3
Sharing keys and values, as in multi-query attention, reduces memory use but restricts specialization. A shared representation must encode several relationships at once, so subtle distinctions can weaken or disappear. section 2.6.3
Model capacity affects what training strategies and signals can be used effectively:
- Larger models can absorb softer probability distributions containing fine-grained relationships; smaller models may be overwhelmed by information they cannot represent. section 8.3.5
- Small models may fail to discover reasoning strategies through reinforcement learning, yet learn those strategies by imitating demonstrations from a larger teacher. section 8.7.3
- Depth and width provide concrete capacity differences: a four-layer, 128-dimensional teacher substantially exceeds a two-layer, 32-dimensional student. section 8.6.1
Toy teacher and student
A teacher with roughly 26,000 parameters uses four 128-dimensional hidden layers, while a student with roughly 8,800 parameters uses two 32-dimensional layers. Both retain the same embedding and output dimensions, isolating hidden-layer capacity as the main difference. section 8.6.1
Figure 8.21 The toy-sized teacher model (4 layers, 128-dim hidden, ~26K parameters) and student model (2 layers, 32-dim hidden, ~8.8K parameters) used in our implementation. The student is 3x smaller but maintains the same input and output dimensions.

Compare the teacher and student architectures side by side to see how depth, width, parameter count, and shared input-output dimensions shape representational capacity. section 8.6.1












