A Transformer is a neural-network architecture organized around attention. Rather than carrying context through a recurrent state one step at a time, it lets sequence elements interact directly to build context-dependent representations. This makes long-range relationships more accessible and permits parallel processing of sequences, helping language models scale. section 3.5.7 section 4.1.1
In language processing, text is divided into tokens. For each token, self-attention weighs information from other permitted tokens and combines it into a new representation. Learned query, key, and value projections separate the roles of seeking relevant context, assessing relevance, and supplying information. Multiple attention heads provide concurrent views of relationships in the same input. section 4.1 section 4.1.3 section 4.8
Figure 4.5 The model turns each token’s representation into three vectors: a query that expresses what it seeks, a key that describes characteristics used to assess relevance, and a value that carries the information it can contribute. Each is produced by its own learned projection. By comparing queries and keys, the attention head produces an attention pattern that reflects how strongly tokens are related in the model’s internal representation, with darker cells in the matrix indicating stronger connections. The values are then combined according to this pattern, giving each token a new representation, known as a hidden state, that integrates contextual information from across the sequence.

The diagram traces how query–key comparisons produce an attention pattern used to combine values into contextualized token representations. section 4.1.3
Attention alone does not encode sequence order, so positional information must be supplied explicitly. A Transformer block combines attention with feedforward layers; residual connections and layer normalization support stable information flow and training in deeper stacks. section 4.8 section 4.1.6
The behavior of a block depends on which interactions are allowed. An encoder can let each input token attend to the whole input. A decoder generates a sequence step by step: a causal mask hides future positions, preventing a prediction from using the answer it is meant to produce. Parallel sequence processing during training therefore does not mean that generation must happen all at once. section 4.1.6
Figure 4.10 Unmasked and masked self-attention. In these matrices, darker cells indicate stronger attention between tokens, while hatched cells mark positions that are masked. In unmasked attention (left), each token can attend to any other token in the sequence. In masked attention (right), future positions are hidden, allowing each token to see only itself and earlier tokens. This causal structure enables step-by-step text generation, where each new token is produced using only what came before.

The attention matrices contrast unrestricted access with the causal mask used for generation. section 4.1.6
The same attention-based approach also supports multimodal systems. Separate encoders can turn text, images, or audio into representations, which a shared Transformer combines through cross-attention. A text representation, for example, can selectively query visual features rather than merging every signal indiscriminately. section 4.5.3
Parallel computation and stable signal flow make large models practical on accelerators, but scale still demands substantial computation, memory, and hardware coordination. More parameters expand potential capacity; they do not guarantee effective learning or performance. Fluent output alone should not be taken as proof of human-like understanding. section 4.2.3 section 4.2.1 section 4.0







































