liveBook · Manning

Transformer block

A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.

How 5 Manning books explain it

Rearchitecting LLMs

A transformer layer, or transformer block, is a processing unit that uses attention and feed-forward components to transform representations.

How it is used
Uses block structure to guide model inspection and depth and width optimization.
How deep it goes
Traces nested forward calls and block components, then explains the effects of removing entire blocks.
Examples
Inspects DistilGPT2's six-block stack and Gemma-3-270M's 18 decoder blocks.
What you can do after
Evaluate model structure and pruning trade-offs by inspecting block counts, components, and baseline behavior.

Read how Rearchitecting LLMs explains transformer block →

Build a DeepSeek Model (From Scratch)

A transformer block is a repeated neural-network unit combining attention, a feed-forward network, normalization, and residual connections.

How it is used
Uses blocks as the repeated model trunk for next-token prediction and as computation units in multi-token prediction modules.
How deep it goes
Explains block data flow and residual structure, then implements a normalized attention and mixture-of-experts block with cache handling.
Examples
Traces a four-token context matrix to logits and implements a DeepSeek-style block in MiniDeepSeek.
What you can do after
Builds a transformer block from component modules and integrates it into a model stack with training and cached inference paths.

Read how Build a DeepSeek Model (From Scratch) explains transformer block →

CUDA for LLMs

A transformer block is a transformer unit containing attention and feed-forward sublayers together with residual connections and layer normalization.

How it is used
Establishes the repeated unit of a transformer baseline for training and CUDA inference.
How deep it goes
Traces a pre-normalized block through attention, feed-forward layers, residual additions, and PyTorch forward calls.
Examples
Uses an eight-block transformer with four heads and activations shaped (16, 64, 128).
What you can do after
Evaluate the block’s forward-pass structure and distinguish its attention and position-wise transformations.

Read how CUDA for LLMs explains transformer block →

Building Specialized AI Systems

A Transformer block is a repeated processing unit containing attention, feed-forward, residual, normalization, and positional components.

How it is used
Frames the fundamental repeated unit used to build Transformer models for generation, representation learning, and sequence-to-sequence tasks.
How deep it goes
Explains the block’s components and input pipeline, without detailing the internal mathematics of each component.
Examples
Uses raw text, subword tokens, integer IDs, learned embeddings, positional embeddings, and encoder-decoder architecture diagrams.
What you can do after
Understand how token representations enter and move through the repeated processing units of Transformer architectures.

Read how Building Specialized AI Systems explains transformer block →

Transformers in Action

A transformer block is a repeated computational unit in a transformer composed of attention and feed-forward components with neural-network connections and normalization.

How it is used
Provides the repeated computational backbone for decoder-only generative models.
How deep it goes
Defines a single block through attention, normalization, residual additions, and an expanded MLP, without deriving their internals.
Examples
Uses a simplified GPT-2-style block with TensorFlow operations and attention-state output.
What you can do after
Build a simplified transformer block and identify its residual computation stages in decoder-only inference.

Read how Transformers in Action explains transformer block →

Also covered in

What to know first

Related concepts

Questions about transformer block

What is transformer block?

A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.

Which Manning books explain transformer block?

Rearchitecting LLMs, Build a DeepSeek Model (From Scratch), CUDA for LLMs, Building Specialized AI Systems and Transformers in Action.

What should I know before learning transformer block?

Manning books that teach transformer block usually introduce transformer architecture, attention, GPT-2 and layer normalization first.

What is transformer block part of?

Transformer block is part of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.

What are the parts of transformer block?

Transformer block includes multi-head attention, feedforward layer, feedforward network, residual connection and attention.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.