liveBook · Manning

Transformer architecture

As explained in Building Specialized AI Systems

A neural-network architecture based primarily on attention mechanisms for processing sequences.

Transformer architecture processes sequences primarily through attention rather than recurrence or convolution. It addresses recurrent models’ slow sequential training and fixed-size memory bottleneck by allowing tokens to interact across a sequence, supporting parallel processing and richer long-range context. section 3.1.1

A standard encoder-decoder Transformer combines these components:

  • Tokenization maps raw text to subword IDs; learned token embeddings are combined with positional embeddings before entering the first block. section 3.1.1
  • Encoder layers use multi-head self-attention and multilayer perceptrons, with residual connections and layer normalization. section 3.1.1
  • Decoder layers use masked self-attention, cross-attention over encoder features, and a linear layer with softmax to produce output probabilities. section 3.1.1
  • Self-attention computes query-key compatibility scores, normalizes them into attention weights, and forms context-aware outputs as weighted value sums. section 3.1.1

Translation pipeline

For language translation, the encoder converts the source sequence into contextual features. The decoder attends to those features while generating the target sequence, avoiding the recurrent encoder’s requirement to compress the entire source into one final hidden state. section 3.1.1

Figure 3.2 Transformer model architecture. It consists of an encoder stack (left) and a decoder stack (right). The encoder processes the input sequence using multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks with residual connections and layer normalization (Add & LN). The decoder processes the target sequence using masked multi-head self-attention (MMSA) and multi-head cross attention (MCA) to integrate encoder features, then projects the output to probabilities through a linear layer and softmax. Source: [9]

Use the architecture figure to trace encoder attention, decoder masking and cross-attention, residual connections, normalization, and output probabilities. section 3.1.1

Figure 3.3 The input pipeline before the Transformer block including the positional embedding. Raw text is tokenized into subword units and mapped to integer IDs, which are then looked up in a learned embedding matrix to produce dense token vectors. Positional embeddings encoding each token's position in the sequence are added to the token embeddings, producing the final input matrix passed to the first Transformer block.

Use the input-pipeline figure to follow text from subword tokenization through integer IDs, embeddings, positional information, and the first Transformer block. section 3.1.1

Attention(Q,K,V)=softmax(QKT{dk})V

Use the attention equation to connect queries, keys, values, softmax normalization, and context-aware outputs. section 3.1.1

How Building Specialized AI Systems covers it

How deep it goes
Explains inputs, encoder-decoder components, attention computation, and architectural motivation, but stops short of implementation code.
Examples
Uses language translation, tokenized subword inputs, attention weights, and encoder-decoder architecture diagrams.
What you can do after
Evaluate how attention, masking, positional information, and encoder-decoder pathways support sequence modeling and later fine-tuning.
How it connects
Connects self-attention, positional embeddings, encoder-decoder models, recurrent neural networks, convolutional models, and fine-tuning.

Also explained in

Compare how 40 books explain transformer architecture →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.