liveBook · Manning

Parameter

As explained in Build a DeepSeek Model (From Scratch)

A model parameter is a learned numerical value that determines part of a machine-learning model's computation.

A parameter, also called a weight, is a learned numerical value that determines part of a machine-learning model’s computation. During pretraining, parameter values are adjusted to capture grammar, facts, reasoning patterns, and statistical relationships between tokens. section 2.1.1

After pretraining and later post-training such as supervised fine-tuning or reinforcement learning, parameters become fixed for inference. The model then uses these stored values to predict the next token rather than continuing to learn. section 2.1.1

Parameter count creates practical deployment trade-offs:

  • A model may activate only part of its parameters for each token, yet still need the complete weight set stored and accessible. section 8.1.1
  • Storage precision and parameter count determine memory requirements: 671 billion parameters require approximately 1342 GB in FP16. section 8.1.1
  • Every generated token incurs computation through the active parameters, while long contexts add KV-cache memory requirements. section 8.1.1

DeepSeek-R1

DeepSeek-R1 has 671 billion total parameters, with 37 billion activated per token. Its full FP16 weight set cannot fit on one H100 or RTX 4090, illustrating why parameter storage constrains deployment. section 8.1.1

Figure 8.2 The deployment gap between DeepSeek-R1 (requiring a server cluster) and real-world targets where reasoning capabilities are needed, from data center GPUs down to mobile devices.

Visualizes the gap between parameter-heavy server models and deployment targets ranging from data-center GPUs to mobile devices. section 8.1.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines learned and fixed parameters, then quantifies their storage and inference costs without detailing parameter-update algorithms.
Examples
Uses GPT-2, DeepSeek-R1, 671-billion-parameter storage, FP16 memory estimates, and NVIDIA GPU comparisons.
What you can do after
Evaluate how parameter count, activation, precision, and storage affect model deployment and inference cost.
How it connects
Connects parameters with pretraining, post-training, inference, model weights, Mixture-of-Experts, and the KV cache.

Also explained in

Compare how 41 books explain parameter →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.