Transformer block
A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.
Rearchitecting LLMs
A transformer layer, or transformer block, is a processing unit that uses attention and feed-forward components to transform representations.
- How it is used
- Uses block structure to guide model inspection and depth and width optimization.
- How deep it goes
- Traces nested forward calls and block components, then explains the effects of removing entire blocks.
- Examples
- Inspects DistilGPT2's six-block stack and Gemma-3-270M's 18 decoder blocks.
- What you can do after
- Evaluate model structure and pruning trade-offs by inspecting block counts, components, and baseline behavior.
Build a DeepSeek Model (From Scratch)
A transformer block is a repeated neural-network unit combining attention, a feed-forward network, normalization, and residual connections.
- How it is used
- Uses blocks as the repeated model trunk for next-token prediction and as computation units in multi-token prediction modules.
- How deep it goes
- Explains block data flow and residual structure, then implements a normalized attention and mixture-of-experts block with cache handling.
- Examples
- Traces a four-token context matrix to logits and implements a DeepSeek-style block in MiniDeepSeek.
- What you can do after
- Builds a transformer block from component modules and integrates it into a model stack with training and cached inference paths.
Read how Build a DeepSeek Model (From Scratch) explains transformer block →
CUDA for LLMs
A transformer block is a transformer unit containing attention and feed-forward sublayers together with residual connections and layer normalization.
- How it is used
- Establishes the repeated unit of a transformer baseline for training and CUDA inference.
- How deep it goes
- Traces a pre-normalized block through attention, feed-forward layers, residual additions, and PyTorch forward calls.
- Examples
- Uses an eight-block transformer with four heads and activations shaped (16, 64, 128).
- What you can do after
- Evaluate the block’s forward-pass structure and distinguish its attention and position-wise transformations.
Building Specialized AI Systems
A Transformer block is a repeated processing unit containing attention, feed-forward, residual, normalization, and positional components.
- How it is used
- Frames the fundamental repeated unit used to build Transformer models for generation, representation learning, and sequence-to-sequence tasks.
- How deep it goes
- Explains the block’s components and input pipeline, without detailing the internal mathematics of each component.
- Examples
- Uses raw text, subword tokens, integer IDs, learned embeddings, positional embeddings, and encoder-decoder architecture diagrams.
- What you can do after
- Understand how token representations enter and move through the repeated processing units of Transformer architectures.
Read how Building Specialized AI Systems explains transformer block →
Transformers in Action
A transformer block is a repeated computational unit in a transformer composed of attention and feed-forward components with neural-network connections and normalization.
- How it is used
- Provides the repeated computational backbone for decoder-only generative models.
- How deep it goes
- Defines a single block through attention, normalization, residual additions, and an expanded MLP, without deriving their internals.
- Examples
- Uses a simplified GPT-2-style block with TensorFlow operations and attention-state output.
- What you can do after
- Build a simplified transformer block and identify its residual computation stages in decoder-only inference.
Read how Transformers in Action explains transformer block →
Also covered in
- Build a Large Language Model (From Scratch)
- Grokking AI Algorithms, Second Edition
- Natural Language Processing in Action, Second Edition
- Machines that ThinkIn early access (MEAP)
- Hugging Face in Action
What to know first
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- GPT-2GPT-2 is a family of autoregressive, decoder-only Transformer language models developed by OpenAI that generate text by predicting successive tokens from preceding context.
- layer normalizationLayer normalization standardizes neural-network activations across specified feature dimensions independently for each example, often using learned scale and bias parameters.
Related concepts
- multi-head attentionhas part · Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
- transformer architecturepart of · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- feedforward networkhas part · A feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.
- residual connectionhas part · A residual connection is a neural-network pathway that carries an earlier representation around one or more transformations so it can be combined with a later computation.
- attentionhas part · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
Questions about transformer block
What is transformer block?
A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.
Which Manning books explain transformer block?
Rearchitecting LLMs, Build a DeepSeek Model (From Scratch), CUDA for LLMs, Building Specialized AI Systems and Transformers in Action.
What should I know before learning transformer block?
Manning books that teach transformer block usually introduce transformer architecture, attention, GPT-2 and layer normalization first.
What is transformer block part of?
Transformer block is part of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
What are the parts of transformer block?
Transformer block includes multi-head attention, feedforward layer, feedforward network, residual connection and attention.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




