A parameter, also called a weight, is a learned numerical value that determines part of a machine-learning model’s computation. During pretraining, parameter values are adjusted to capture grammar, facts, reasoning patterns, and statistical relationships between tokens. section 2.1.1
After pretraining and later post-training such as supervised fine-tuning or reinforcement learning, parameters become fixed for inference. The model then uses these stored values to predict the next token rather than continuing to learn. section 2.1.1
Parameter count creates practical deployment trade-offs:
- A model may activate only part of its parameters for each token, yet still need the complete weight set stored and accessible. section 8.1.1
- Storage precision and parameter count determine memory requirements: 671 billion parameters require approximately 1342 GB in FP16. section 8.1.1
- Every generated token incurs computation through the active parameters, while long contexts add KV-cache memory requirements. section 8.1.1
DeepSeek-R1
DeepSeek-R1 has 671 billion total parameters, with 37 billion activated per token. Its full FP16 weight set cannot fit on one H100 or RTX 4090, illustrating why parameter storage constrains deployment. section 8.1.1
Figure 8.2 The deployment gap between DeepSeek-R1 (requiring a server cluster) and real-world targets where reasoning capabilities are needed, from data center GPUs down to mobile devices.

Visualizes the gap between parameter-heavy server models and deployment targets ranging from data-center GPUs to mobile devices. section 8.1.1








































