liveBook · Manning

Transformer block

As explained in Building Specialized AI Systems

A Transformer block is a repeated processing unit containing attention, feed-forward, residual, normalization, and positional components.

A Transformer block is a repeated processing unit that transforms token representations before passing them to the next layer. Its essential components are token embeddings, positional information, self-attention, a position-wise feed-forward network, residual shortcuts, and normalization. section 3.1.1

A block supports the following processing flow:

  • Converts tokenized text into numerical vectors and combines them with positional information so word order is represented. section 3.1.1
  • Applies self-attention to connect information across the sequence, then uses a position-wise feed-forward network for further transformation. section 3.1.1
  • Uses residual shortcuts and normalization while passing refined representations to subsequent blocks. section 3.1.1
Figure 3.3 The input pipeline before the Transformer block including the positional embedding. Raw text is tokenized into subword units and mapped to integer IDs, which are then looked up in a learned embedding matrix to produce dense token vectors. Positional embeddings encoding each token's position in the sequence are added to the token embeddings, producing the final input matrix passed to the first Transformer block.

Trace the path from raw text through subword tokenization, integer IDs, learned embeddings, and positional embeddings to the first Transformer block. section 3.1.1

How Building Specialized AI Systems covers it

How deep it goes
Explains the block’s components and input pipeline, without detailing the internal mathematics of each component.
Examples
Uses raw text, subword tokens, integer IDs, learned embeddings, positional embeddings, and encoder-decoder architecture diagrams.
What you can do after
Understand how token representations enter and move through the repeated processing units of Transformer architectures.
How it connects
Connects tokenization, token embeddings, positional embeddings, self-attention, feed-forward networks, residual connections, and normalization.

Also explained in

Compare how 5 books explain transformer block →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.