Transformer block

As explained in Transformers in Action

A transformer block is a repeated computational unit in a transformer composed of attention and feed-forward components with neural-network connections and normalization.

A transformer block is the repeated computational unit that forms a decoder-only model’s backbone. Each block combines attention, a feed-forward network, residual connections, and normalization; stacking blocks increases capacity and expressiveness while also affecting inference latency. section 3.2

Figure 3.1 Abstracted decoder-only architecture

Use the architecture figure to relate one block to the sequential stack that constitutes a decoder-only model. section 3.2

How Transformers in Action covers it

How deep it goes

Defines a single block through attention, normalization, residual additions, and an expanded MLP, without deriving their internals.

Examples

Uses a simplified GPT-2-style block with TensorFlow operations and attention-state output.

What you can do after

Build a simplified transformer block and identify its residual computation stages in decoder-only inference.

How it connects

Connects attention, feed-forward networks, residual connections, normalization, autoregressive decoding, and KV caching.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• transformer architecturepart of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• feedforward networkhas part · A feedforward network is a neural network or subnetwork in which information flows from inputs toward outputs through successive computations without recurrent connections or cycles.
• residual connectionhas part · A residual connection is a neural-network pathway that carries an earlier representation around one or more transformations to a later computation, where it can be combined with the transformed representation.
• layer normalizationhas part · Layer normalization standardizes neural-network activations across specified feature dimensions for each example independently, often applying learned scale and bias parameters.
• attentionhas part · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.