liveBook · Manning

Expressive power

As explained in Build a DeepSeek Model (From Scratch)

Representational capacity is the ability of a model architecture to represent functions, patterns, or input-output relationships.

Expressive power is an architecture’s ability to represent diverse functions, patterns, or input-output relationships. In multi-head attention, independent key and value projections let heads specialize in different relationships and blend them into richer context vectors. section 2.6.3

Sharing keys and values, as in multi-query attention, reduces memory use but restricts specialization. A shared representation must encode several relationships at once, so subtle distinctions can weaken or disappear. section 2.6.3

Model capacity affects what training strategies and signals can be used effectively:

  • Larger models can absorb softer probability distributions containing fine-grained relationships; smaller models may be overwhelmed by information they cannot represent. section 8.3.5
  • Small models may fail to discover reasoning strategies through reinforcement learning, yet learn those strategies by imitating demonstrations from a larger teacher. section 8.7.3
  • Depth and width provide concrete capacity differences: a four-layer, 128-dimensional teacher substantially exceeds a two-layer, 32-dimensional student. section 8.6.1

Toy teacher and student

A teacher with roughly 26,000 parameters uses four 128-dimensional hidden layers, while a student with roughly 8,800 parameters uses two 32-dimensional layers. Both retain the same embedding and output dimensions, isolating hidden-layer capacity as the main difference. section 8.6.1

Figure 8.21 The toy-sized teacher model (4 layers, 128-dim hidden, ~26K parameters) and student model (2 layers, 32-dim hidden, ~8.8K parameters) used in our implementation. The student is 3x smaller but maintains the same input and output dimensions.

Compare the teacher and student architectures side by side to see how depth, width, parameter count, and shared input-output dimensions shape representational capacity. section 8.6.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Compares attention representations, parameterized teacher-student architectures, and capacity limits in temperature scaling and reinforcement learning.
Examples
Uses specialized attention heads, toy teacher-student networks, temperature values from 1 to 10, and model sizes from 1.5B to 671B parameters.
What it assumes
Assumes familiarity with attention, teacher-student models, temperature scaling, and reinforcement learning while grounding capacity through concrete architectures.
What you can do after
Evaluate how architecture and parameter count constrain specialization, probability-signal absorption, and reasoning discovery.
How it connects
Connects multi-head attention, multi-query attention, knowledge distillation, temperature scaling, and reinforcement learning.

Also explained in

Compare how 13 books explain expressive power →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.