Dense model
A dense model is a neural network that uses its applicable parameters and computational components for each input rather than routing computation to a sparse subset.
Transformers in Action
A dense model is a neural model in which all relevant parameters and computational components are activated for every input.
- How it is used
- Uses full parameter activation as the baseline for comparing transformer computation with conditional mixture-of-experts routing.
- How deep it goes
- Defines activation behavior and identifies feed-forward and attention processing, stopping short of dense-layer implementation details.
- Examples
- Uses BERT and GPT as examples of traditional dense transformer models.
- What you can do after
- Evaluate dense and mixture-of-experts architectures by comparing parameter activation, computational cost, and scaling behavior.
Self-Hosting LLMs at Scale
A dense model is a neural network in which the model’s parameters participate in computation for each input, rather than selecting a sparse subset.
- How it is used
- Uses dense models as a baseline for self-hosted model selection and capacity decisions.
- How deep it goes
- Explains token-level activation, transformer layers, and memory trade-offs, without implementing the architecture.
- Examples
- Compares a 70-billion-parameter dense model with a 200-billion-parameter MoE model activating 20 billion parameters per token.
- What you can do after
- Evaluate dense and MoE models by their memory and per-token computation requirements.
Also covered in
- Machines that ThinkIn early access (MEAP)
Related concepts
- mixture of expertscontrasts with · A mixture-of-experts model is a neural-network architecture with specialized subnetworks and a routing mechanism that selects or combines experts for each input.
- sparse upcyclingcontrasts with · Sparse Upcycling is a method for converting a pretrained dense model into a sparse mixture-of-experts model, commonly by duplicating feed-forward components as experts and further training the resulting architecture.
- linear layeruses · A linear layer is a neural-network layer that applies a learned affine transformation to its input, typically using weights and a bias.
Questions about dense model
What is dense model?
A dense model is a neural network that uses its applicable parameters and computational components for each input rather than routing computation to a sparse subset.
Which Manning books explain dense model?
Transformers in Action and Self-Hosting LLMs at Scale.
What is the difference between dense model and mixture of experts?
A dense model is a neural network that uses its applicable parameters and computational components for each input rather than routing computation to a sparse subset. A mixture-of-experts model is a neural-network architecture with specialized subnetworks and a routing mechanism that selects or combines experts for each input.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

