Transformer architecture processes sequences primarily through attention rather than recurrence or convolution. It addresses recurrent models’ slow sequential training and fixed-size memory bottleneck by allowing tokens to interact across a sequence, supporting parallel processing and richer long-range context. section 3.1.1
A standard encoder-decoder Transformer combines these components:
- Tokenization maps raw text to subword IDs; learned token embeddings are combined with positional embeddings before entering the first block. section 3.1.1
- Encoder layers use multi-head self-attention and multilayer perceptrons, with residual connections and layer normalization. section 3.1.1
- Decoder layers use masked self-attention, cross-attention over encoder features, and a linear layer with softmax to produce output probabilities. section 3.1.1
- Self-attention computes query-key compatibility scores, normalizes them into attention weights, and forms context-aware outputs as weighted value sums. section 3.1.1
Translation pipeline
For language translation, the encoder converts the source sequence into contextual features. The decoder attends to those features while generating the target sequence, avoiding the recurrent encoder’s requirement to compress the entire source into one final hidden state. section 3.1.1
Figure 3.2 Transformer model architecture. It consists of an encoder stack (left) and a decoder stack (right). The encoder processes the input sequence using multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks with residual connections and layer normalization (Add & LN). The decoder processes the target sequence using masked multi-head self-attention (MMSA) and multi-head cross attention (MCA) to integrate encoder features, then projects the output to probabilities through a linear layer and softmax. Source: [9]

Use the architecture figure to trace encoder attention, decoder masking and cross-attention, residual connections, normalization, and output probabilities. section 3.1.1
Figure 3.3 The input pipeline before the Transformer block including the positional embedding. Raw text is tokenized into subword units and mapped to integer IDs, which are then looked up in a learned embedding matrix to produce dense token vectors. Positional embeddings encoding each token's position in the sequence are added to the token embeddings, producing the final input matrix passed to the first Transformer block.

Use the input-pipeline figure to follow text from subword tokenization through integer IDs, embeddings, positional information, and the first Transformer block. section 3.1.1
Use the attention equation to connect queries, keys, values, softmax normalization, and context-aware outputs. section 3.1.1







































