liveBook · Manning

Transformer architecture

As explained in Understanding AI

A Transformer is a neural-network architecture that uses attention mechanisms to model relationships among sequence elements, often in parallel.

How can one AI system answer a question, summarize a document, translate a sentence, and help with code? A major part of the answer is the Transformer: a neural-network architecture built around attention. Rather than following a long list of hand-written rules, it learns patterns from large amounts of data and uses context to make predictions. section 1.0 section 1.5

Attention is the core idea. When predicting what comes next, a Transformer can focus directly on the parts of the input that matter most. This helps it connect words or other elements that are far apart, instead of relying only on information passed forward one step at a time. section 1.3.1 section 1.5

A distant relationship in a sentence

Imagine reading a sentence whose important clue appears near the beginning but affects a word much later. Earlier sequence models process the text in order, carrying information from one step to the next. A Transformer can compare relationships across the full sequence at once, making the earlier clue available directly when it is useful. section 1.3.1

This design changes the practical side of training as well. Because relationships across a sequence can be evaluated simultaneously, Transformer models can use GPUs and distributed computation efficiently. That makes training faster and easier to scale than earlier sequential approaches, helping make very large models practical. section 1.3.1

As models grow and learn from enormous collections of text, the same Transformer-based model can generalize across tasks such as:

  • answering questions and summarizing documents section 1.3.1
  • generating code, translating languages, and following instructions section 1.3.1
  • working with text alongside images, audio, video, and code in multimodal systems section 1.3.2

A Transformer is not a human-like thinker. Even when its response is useful and well structured, its behavior is grounded in learned statistical patterns and prediction. Its power comes from attention, data, computation, and scale—not from human understanding. section 1.0 section 1.5

How Understanding AI covers it

How deep it goes
Explains attention, parallel sequence processing, scalable training, and broad task generalization without detailing internal computations.
Examples
Uses distant relationships in text and multimodal actions such as image questions, screenshot debugging, and voice-text interaction.
What you can do after
Evaluate/understand why attention-based models scale beyond narrow tasks while still relying on statistical patterns rather than human understanding.
How it connects
Connects attention mechanisms with GPUs, distributed computation, large language models, multimodal systems, and pattern prediction.

Also explained in

Compare how 40 books explain transformer architecture →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Understanding AI. The section links point to the parts of the book it draws on.