liveBook · Manning

Dense model

A dense model is a neural network that uses its applicable parameters and computational components for each input rather than routing computation to a sparse subset.

How 2 Manning books explain it

Transformers in Action

A dense model is a neural model in which all relevant parameters and computational components are activated for every input.

How it is used
Uses full parameter activation as the baseline for comparing transformer computation with conditional mixture-of-experts routing.
How deep it goes
Defines activation behavior and identifies feed-forward and attention processing, stopping short of dense-layer implementation details.
Examples
Uses BERT and GPT as examples of traditional dense transformer models.
What you can do after
Evaluate dense and mixture-of-experts architectures by comparing parameter activation, computational cost, and scaling behavior.

Read how Transformers in Action explains dense model →

Self-Hosting LLMs at Scale

A dense model is a neural network in which the model’s parameters participate in computation for each input, rather than selecting a sparse subset.

How it is used
Uses dense models as a baseline for self-hosted model selection and capacity decisions.
How deep it goes
Explains token-level activation, transformer layers, and memory trade-offs, without implementing the architecture.
Examples
Compares a 70-billion-parameter dense model with a 200-billion-parameter MoE model activating 20 billion parameters per token.
What you can do after
Evaluate dense and MoE models by their memory and per-token computation requirements.

Read how Self-Hosting LLMs at Scale explains dense model →

Also covered in

Related concepts

Questions about dense model

What is dense model?

A dense model is a neural network that uses its applicable parameters and computational components for each input rather than routing computation to a sparse subset.

Which Manning books explain dense model?

Transformers in Action and Self-Hosting LLMs at Scale.

What is the difference between dense model and mixture of experts?

A dense model is a neural network that uses its applicable parameters and computational components for each input rather than routing computation to a sparse subset. A mixture-of-experts model is a neural-network architecture with specialized subnetworks and a routing mechanism that selects or combines experts for each input.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.