Dense model

As explained in Transformers in Action

A dense model is a neural model in which all relevant parameters and computational components are activated for every input.

A dense model activates all model parameters for every input token. In transformer architectures, every feed-forward layer and attention head processes every token, producing a high, uniform computational cost. section 3.5.1

Dense computation has practical consequences:

• It applies the same full set of computational components regardless of whether each component contributes meaningfully to the output. section 3.5.1
• As model size increases, its full activation pattern becomes a limiting factor for training and inference. section 3.5.1
• Mixture-of-experts models contrast with dense models by activating only selected expert feed-forward networks for each token. section 3.5.1

Figure 3.4 Simplified MoE architecture. Only a few experts are active per input, making inference more efficient.

Compare the dense pattern of activating all components with sparse mixture-of-experts routing, where only a few experts process each input. section 3.5.1

How Transformers in Action covers it

How deep it goes

Defines activation behavior and identifies feed-forward and attention processing, stopping short of dense-layer implementation details.

Examples

Uses BERT and GPT as examples of traditional dense transformer models.

What you can do after

Evaluate dense and mixture-of-experts architectures by comparing parameter activation, computational cost, and scaling behavior.

How it connects

Connects dense models to transformers, feed-forward networks, attention mechanisms, mixture-of-experts, and conditional computation.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• mixture of expertscontrasts with · A mixture-of-experts model is a neural-network architecture with specialized expert subnetworks and a routing mechanism that selects or combines experts for each input.
• linear layeruses · A linear layer is a neural-network layer that maps input features to outputs by a learned affine transformation, typically connecting each output to every input feature.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.