liveBook · Manning

Parameter count

As explained in Build a DeepSeek Model (From Scratch)

Parameter count is the number of learned numerical parameters in a machine-learning model.

Parameter count measures a model’s learned numerical parameters. It helps compare model scale and, for dense transformer feed-forward networks, relates to the computation used for each input token. section 4.1.1 section 8.6.1

Parameter totals help make architectural trade-offs concrete:

  • Dense FFNs activate all their parameters for each input token, contributing to training cost and inference latency. section 4.1.1
  • In a toy distillation setup, the teacher has roughly 26,000 parameters and the student roughly 8,800; the student is about three times smaller. section 8.6.1
Figure 4.2 The standard Feed-Forward Network (FFN) in a Transformer block, featuring an expansion-contraction architecture.

Shows the expansion-contraction structure of a dense transformer FFN, the architecture whose per-token parameter use motivates the cost comparison. section 4.1.1

Figure 8.21 The toy-sized teacher model (4 layers, 128-dim hidden, ~26K parameters) and student model (2 layers, 32-dim hidden, ~8.8K parameters) used in our implementation. The student is 3x smaller but maintains the same input and output dimensions.

Compares the toy teacher and student architectures alongside their approximate parameter totals. section 8.6.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Compares parameter totals and architectural implications, but supplies no general procedure for counting parameters.
Examples
Uses dense transformer FFNs and a 26,000- versus 8,800-parameter teacher–student pair.
What you can do after
Evaluate model-size trade-offs by relating parameter totals to dense computation costs and teacher–student differences.
How it connects
Connects parameter count to feed-forward networks, transformer models, and knowledge distillation.

Also explained in

Compare how 6 books explain parameter count →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.