A Transformer block is a repeated processing unit that transforms token representations before passing them to the next layer. Its essential components are token embeddings, positional information, self-attention, a position-wise feed-forward network, residual shortcuts, and normalization. section 3.1.1
A block supports the following processing flow:
- Converts tokenized text into numerical vectors and combines them with positional information so word order is represented. section 3.1.1
- Applies self-attention to connect information across the sequence, then uses a position-wise feed-forward network for further transformation. section 3.1.1
- Uses residual shortcuts and normalization while passing refined representations to subsequent blocks. section 3.1.1
Figure 3.3 The input pipeline before the Transformer block including the positional embedding. Raw text is tokenized into subword units and mapped to integer IDs, which are then looked up in a learned embedding matrix to produce dense token vectors. Positional embeddings encoding each token's position in the sequence are added to the token embeddings, producing the final input matrix passed to the first Transformer block.

Trace the path from raw text through subword tokenization, integer IDs, learned embeddings, and positional embeddings to the first Transformer block. section 3.1.1




