Transformer architecture
A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
Learn Generative AI with PyTorch
The internal design and structural organization of a Transformer model built around decoder blocks or self-attention components.
- How it is used
- Supports sequence-to-sequence translation and next-token generation through encoder–decoder or decoder-only arrangements.
- How deep it goes
- Works through layer structure, attention tensor shapes, and a five-component implementation, then uses the trained model for translation.
- Examples
- Uses English-to-French translation, a shaped multihead-attention example, and a scaled-down GPT-style model.
- What you can do after
- Build it from scratch as an encoder–decoder translation model, then apply its trained encoder and decoder to translate English phrases.
Read how Learn Generative AI with PyTorch explains transformer architecture →
Natural Language Processing in Action, Second Edition
A transformer is a neural network architecture that uses attention mechanisms to process sequences and model relationships among their elements.
- How it is used
- Supports NLP tasks through stackable sequence processing, with translation serving as a practical implementation case.
- How deep it goes
- Explains recurrence, tokenization, attention, positional encoding, and layer shapes, then works through a PyTorch translation model.
- Examples
- Uses a German–English translation pipeline, tensor-shape checks, sample translations, and decoder attention visualizations.
- What you can do after
- Apply it via PyTorch’s nn.Transformer to train, evaluate, and inspect a translation model.
Read how Natural Language Processing in Action, Second Edition explains transformer architecture →
Sutskever's List
A Transformer is a neural network architecture based primarily on attention mechanisms for modeling relationships among sequence elements.
- How it is used
- Frames the architectural shift from recurrent translation models to attention-based translation and language models.
- How deep it goes
- Explains direct token connections and identifies major components, while pointing to an annotated implementation for line-by-line detail.
- Examples
- Uses a “cheese”–“mice” dependency, translation BLEU scores, GPT-2, and a GPipe-trained Transformer.
- What you can do after
- Evaluate/understand why direct self-attention improved translation and training, while distinguishing plausible language generation from demonstrated understanding.
Read how Sutskever's List explains transformer architecture →
Deep Learning with Python, Third Edition
A neural-network architecture based on attention mechanisms, used here as the core architecture for large language models.
- How it is used
- Uses attention-based sequence modeling for translation, text classification, and next-token prediction.
- How deep it goes
- Builds encoder–decoder translation blocks and tests positional embeddings, masks, and training behavior; does not derive the architecture mathematically.
- Examples
- Works through English-to-Spanish translation and compares a position-blind Transformer with one using learned positional embeddings.
- What you can do after
- Build it from scratch in Keras as an encoder–decoder translation model, then diagnose missing word-order information.
Read how Deep Learning with Python, Third Edition explains transformer architecture →
Build a Text-to-Image Generator (from Scratch)
A transformer architecture is a neural-network architecture that uses attention mechanisms to process relationships among elements in a sequence or other structured input.
- How it is used
- Uses translation to establish an attention-based architecture before applying sequence prediction to text-to-image generation.
- How deep it goes
- Builds an encoder–decoder translator from embeddings through attention, masks, training, and prediction, then extends the pipeline to image tokens.
- Examples
- Works through German-to-English translation and image generation from a panda prompt, including intermediate image-token outputs.
- What you can do after
- Build it from scratch in PyTorch as a trained encoder–decoder translator, then build a transformer-based text-to-image generator.
Read how Build a Text-to-Image Generator (from Scratch) explains transformer architecture →
Deep Learning with R, Third Edition
A Transformer is a neural-network architecture designed to process sets or sequences using attention-based contextual representations.
- How it is used
- Uses attention-based sequence modeling for English-to-Spanish translation, then extends to pretrained text classification.
- How deep it goes
- Works through encoder blocks, masking, encoder–decoder assembly, and a diagnosed word-order failure using implementation code.
- Examples
- Compares translation accuracy without and with learned position embeddings, then generates English-to-Spanish translations.
- What you can do after
- Build it from scratch in R/Keras as an encoder–decoder translation model with learned positional embeddings.
Read how Deep Learning with R, Third Edition explains transformer architecture →
How Large Language Models Work
A transformer is a neural-network architecture that uses attention mechanisms to process relationships among elements in a sequence.
- How it is used
- Explains how language models turn token sequences into predictions through embeddings, attention, repeated computation, and sampling.
- How deep it goes
- Explains layers, attention, prediction types, and scaling behavior, but stops short of building an LLM from scratch or exposing proprietary details.
- Examples
- Uses Python dictionaries, preceding words, next-token prediction, translation, summarization, and a component diagram.
- What you can do after
- Evaluate/understand transformer outputs, model variants, sampling behavior, and practical deployment trade-offs.
Read how How Large Language Models Work explains transformer architecture →
AI-Powered Search
A transformer is a neural-network architecture that models relationships among sequence elements using attention and can encode and decode representations.
- How it is used
- Supplies contextual embeddings for semantic retrieval, natural-language autocomplete, and search ranking.
- How deep it goes
- Explains attention, encoding, decoding, and a query-to-index retrieval pipeline without specifying model training internals.
- Examples
- Uses contrasting capital queries, English–Armenian greetings, and a transformer-encoded autocomplete pipeline.
- What you can do after
- Apply it via a pretrained transformer to encode content and queries for nearest-neighbor search.
Read how AI-Powered Search explains transformer architecture →
Transformers in Action
A transformer is a neural network architecture that processes sequential and other structured data using attention mechanisms.
- How it is used
- Supports sequence tasks such as translation and text generation by using attention to capture contextual relationships without recurrent processing.
- How deep it goes
- Explains encoder–decoder flow and names attention, feed-forward networks, residual connections, layer normalization, and positional encoding; the supplied overview does not derive their implementations.
- Examples
- Illustrates encoder–decoder flow with English-to-French translation and multihead attention with the sentiment of “The movie was not bad.”
- What you can do after
- Evaluate/understand it to choose between encoder–decoder and decoder-only models and weigh pretrained-model convenience against resource demands.
Read how Transformers in Action explains transformer architecture →
Deep Learning with PyTorch, Second Edition
A neural network architecture for sequence modeling and content understanding based on attention rather than recurrence or convolution.
- How it is used
- Uses attention-based sequence modeling to explain text generation, content understanding, and translation.
- How deep it goes
- Builds a PyTorch decoder from attention and embeddings, while leaving implementation of the full encoder–decoder transformer as an exercise.
- Examples
- Demonstrates character-level name generation and an encoder–decoder translation diagram.
- What you can do after
- Build it from scratch as a PyTorch decoder for character-level generation; explore encoder–decoder implementation as a further exercise.
Read how Deep Learning with PyTorch, Second Edition explains transformer architecture →
LLMs in Production
A transformer architecture is a neural-network architecture that models sequence relationships primarily through attention rather than recurrence or convolution.
- How it is used
- Frames the attention-based architecture behind LLMs and the choice between combined and single-purpose arrangements.
- How deep it goes
- Explains encoder–decoder roles, parallelism and training trade-offs, but does not derive attention calculations in the supplied passages.
- Examples
- Uses translation, summarization and T5 to illustrate the full architecture, alongside a PyTorch training-from-scratch listing.
- What you can do after
- Evaluate/understand architecture choices against task fit, parallelism, data needs and training memory.
Read how LLMs in Production explains transformer architecture →
Grokking AI Applications
A transformer is a neural-network architecture for sequential data that uses attention to relate elements of an input sequence.
- How it is used
- Explains how GPT-style systems process ordered text and generate responses through attention, deep neural networks, and token prediction.
- How deep it goes
- Defines attention and layered processing with an example, but stops short of mathematical or implementation-level internals.
- Examples
- Uses GPT-3, the question about Italy's capital, token-by-token generation, and diagrams of GPT architecture and training.
- What you can do after
- Understand how attention and layered neural networks support GPT-style sequence processing and text generation.
Read how Grokking AI Applications explains transformer architecture →
The AI Pocket Book
A neural network architecture that contextualizes each token using surrounding tokens rather than compressing the whole prompt into one summary vector.
- How it is used
- Explains how current LLMs turn token embeddings into context-sensitive representations for next-token prediction.
- How deep it goes
- Defines contextualization and attention-based updates, but stops short of exact calculations and implementation details.
- Examples
- Uses dog-versus-tree “bark” prompts, token embeddings, attention scores, and a compact GPT-2 implementation as concrete illustrations.
- What you can do after
- Evaluate/understand why transformers preserve contextual information better than single-summary LSTM representations.
Read how The AI Pocket Book explains transformer architecture →
Time Series Forecasting Using Foundation Models
A neural-network architecture built from stacked attention and feed-forward processing components, typically organized into encoder and decoder parts.
- How it is used
- Uses attention-based sequence processing to explain and support time-series foundation-model forecasting.
- How deep it goes
- Explains embeddings, positional encoding, encoder-decoder flow, attention variants, masking, and projection, but defers attention mathematics.
- Examples
- Uses electricity-demand forecasting with temperature, plus trend and seasonal patterns as attention-learning examples.
- What you can do after
- Evaluate transformer-based forecasters and diagnose architectural limitations involving exogenous features, multiple series, and fine-tuning.
Read how Time Series Forecasting Using Foundation Models explains transformer architecture →
Machine Learning Algorithms in Depth
A transformer model is a neural network architecture that uses attention-based mechanisms to model contextual relationships in structured sequences or other data.
- How it is used
- Uses transformer architecture to connect attention mechanisms with an implemented text-classification task.
- How deep it goes
- Explains query–key–value weighting and major architectural components, then provides a classifier implementation using Keras/TensorFlow layers.
- Examples
- Uses a translation attention heatmap, a transformer architecture diagram, and text-classifier training and validation plots.
- What you can do after
- Apply it via Keras/TensorFlow to text classification and inspect training and validation loss and accuracy.
Read how Machine Learning Algorithms in Depth explains transformer architecture →
Inside Deep Learning: Math, Algorithms, Models
A sequence model built from positional encoding and multiheaded attention, used here as the standard original design for NLP and other tasks.
- How it is used
- Replaces RNN-based sequence processing when accuracy and scalable parallel computation justify higher training cost.
- How deep it goes
- Defines positional encoding, multiheaded attention, encoder-decoder blocks, residual paths, and future masking, without deriving their mathematics.
- Examples
- Uses sentiment classification, machine translation, PyTorch encoder code, and training-accuracy comparisons with GRU, CNN, and attention models.
- What you can do after
- Build a basic PyTorch transformer classifier and recognize when autoregressive attention requires future-token masking.
Read how Inside Deep Learning: Math, Algorithms, Models explains transformer architecture →
Grokking AI Algorithms, Second Edition
The Transformer is a neural-network architecture based primarily on attention mechanisms for processing sequences and modeling context.
- How it is used
- Uses attention-based sequence processing to power language models that generate text, translate languages, answer questions, and write code.
- How deep it goes
- Explains self-attention, block components, architecture alternatives, and hyperparameters, but stops short of equations or implementation details.
- Examples
- Uses a toy language model with 34 tokens, 32-token context, two layers, one attention head, and small feed-forward networks.
- What you can do after
- Evaluate Transformer designs by comparing vocabulary, context, width, depth, attention-head, and feed-forward-size choices.
Read how Grokking AI Algorithms, Second Edition explains transformer architecture →
Hugging Face in Action
The Transformer architecture is a neural-network architecture based on attention mechanisms for processing sequences and other structured inputs.
- How it is used
- Deploys Transformer models through high-level pipelines for practical NLP tasks such as classification, generation, and summarization.
- How deep it goes
- Explains self-attention, embeddings, positions, and encoder-decoder roles briefly, while stopping short of detailed Transformer internals.
- Examples
- Demonstrates IMDb sentiment analysis with a pretrained DistilBERT model and Hugging Face’s pipeline API.
- What you can do after
- Apply it via Hugging Face pipelines to run pretrained models for classification and other NLP workflows.
Read how Hugging Face in Action explains transformer architecture →
Domain-Specific Small Language Models
A neural network architecture for sequence modeling built around self-attention and parallel processing, underlying both large and small language models.
- How it is used
- Frames modern text generation and prediction around parallel sequence processing, self-attention, embeddings, and deployment tradeoffs.
- How deep it goes
- Explains motivation, self-attention, embeddings, and encoder-decoder roles, but stops short of mathematical attention internals.
- Examples
- Uses GPT-Neo large, Hugging Face Transformers, Accelerate, and companion Colab notebooks to ground inference discussion.
- What you can do after
- Evaluate Transformer variants and choose pretrained models with clearer expectations for parallelism, inference cost, and deployment.
Read how Domain-Specific Small Language Models explains transformer architecture →
Introduction to Generative AI, Second Edition
A transformer is a neural-network architecture for sequence modeling that uses attention, especially self-attention, rather than recurrence.
- How it is used
- Explains the architectural shift that enabled modern language models and generative AI applications.
- How deep it goes
- Defines attention and self-attention through intuition and history, but stops short of implementation details, mathematics, or training procedures.
- Examples
- Uses a cat-and-mouse sentence, translation benchmarks, GPT, and attention visualizations to illustrate the architecture.
- What you can do after
- Evaluate/understand it by relating parallel self-attention to long-range context, scalable training, and LLM applications.
Read how Introduction to Generative AI, Second Edition explains transformer architecture →
Build AI-Enhanced Web Apps
A transformer model is a neural-network architecture that uses attention mechanisms to model contextual relationships in sequence data.
- How it is used
- Supports generative AI and language-model applications requiring global context, contextual relationships, and large-scale sequence processing.
- How deep it goes
- Defines self-attention, parallel processing, and contextual relationships with an example, but stops short of implementation details.
- Examples
- Uses pronoun resolution in a sentence to demonstrate how attention captures relationships separated by multiple words.
- What you can do after
- Evaluate transformer suitability for applications involving global context, language modeling, and sequence relationships.
Read how Build AI-Enhanced Web Apps explains transformer architecture →
Build a Large Language Model (From Scratch)
a neural network architecture based on self-attention mechanisms that enables processing of sequential data and underpins many large language models.
- How it is used
- Organizes language processing into contextual encoding and sequential generation for translation and large language models.
- How deep it goes
- Defines encoder, decoder, and self-attention, but defers self-attention implementation and input preprocessing to later treatment.
- Examples
- Uses English-to-German translation, including generating “Beispiel” after “Das ist ein.”
- What you can do after
- Evaluate/understand how encoder-decoder flow and self-attention support translation and GPT-like text generation.
Read how Build a Large Language Model (From Scratch) explains transformer architecture →
Build a Website with ChatGPT
A transformer is a neural-network architecture that processes sequences using attention mechanisms to model relationships among their components.
- How it is used
- Explains why GPT can turn plain-English website requests into relevant generated code.
- How deep it goes
- Defines transformer as prioritizing important request components but stops short of describing attention layers or implementation.
- Examples
- Uses web-page requests and the prompt-copy-save-deploy cycle as practical examples.
- What you can do after
- Apply it via ChatGPT by writing prompts, copying generated code, saving an HTML file, and deploying it.
Read how Build a Website with ChatGPT explains transformer architecture →
Data Analysis with LLMs
The Transformer is a neural-network architecture designed to model relationships in sequences, especially sequences with variable-length inputs or outputs.
- How it is used
- Supports learning tasks with variable-length inputs or outputs, especially text, and underlies generative AI.
- How deep it goes
- Defines its sequence-learning role and broad prevalence without describing internal components or implementation.
- Examples
- Illustrates variable-length sequences with text documents and names document summarization and code writing as GPT tasks.
Read how Data Analysis with LLMs explains transformer architecture →
Generative AI in Action
The Transformer architecture is a neural-network architecture based primarily on attention mechanisms for processing sequences.
- How it is used
- Enables foundational models to understand and generate language for summarization, translation, classification, question answering, and content generation.
- How deep it goes
- Explains self-attention, contextual dependencies, parallel computation, and next-item prediction, but stops short of mathematical or layer-level internals.
- Examples
- Uses the completion “The cat sat on the...” to illustrate attention-weighted prediction of “mat,” alongside GPT and BERT as Transformer models.
- What you can do after
- Evaluate Transformer use for language generation while recognizing its substantial computational demands for training and use.
Read how Generative AI in Action explains transformer architecture →
Coding with AI
A Transformer is a neural-network architecture that processes sequence elements using attention and can process them in parallel.
- How it is used
- Underpins large language models for coding tools, chatbots, translation, and content generation.
- How deep it goes
- Explains input identifiers, positions, self-attention, parallel processing, and scaling, without detailing layer internals.
- Examples
- Uses “Jane helps Joe,” “The dog chased the cat,” and a figure showing positions and relationships.
- What you can do after
- Evaluate/understand it by tracing how prompts become context-sensitive representations for language applications.
Optimization Algorithms: AI techniques for design, planning, and control problems
A transformer architecture is a neural-network architecture that processes sequences primarily through attention mechanisms rather than recurrence.
- How it is used
- Processes sequential data such as text and translation by prioritizing relevant input parts during prediction.
- How deep it goes
- Defines attention intuitively as relevance scores but does not detail Transformer layers, equations, or training.
- Examples
- Uses word-by-word reading, long sentences, translation, and attention coefficients as explanatory examples.
- What you can do after
- Understand how attention can reduce context loss and confusion in long or complex sequences.
Essential GraphRAG
A transformer architecture is a neural-network architecture that processes sequences using attention-based representations and supports efficient language modeling.
- How it is used
- Explains the architecture underlying LLM text processing, generation, and instruction-following behavior.
- How deep it goes
- Defines the architecture’s role and training objective but stops short of explaining internal mechanisms or implementation details.
- Examples
- Uses next-word completion, “Never gonna” to “give you up,” and haiku generation as concrete LLM examples.
- What you can do after
- Evaluate/understand how transformer-based LLMs generate text and why external retrieval can improve factual responses.
Read how Essential GraphRAG explains transformer architecture →
Related concepts
- attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
- encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
- decodehas part · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
- Vision Transformerincludes · A Vision Transformer is a Transformer-based neural-network architecture that processes images as sequences of tokenized regions using attention.
- model architecturekind of · The structural organization of a computational system’s components, relationships, and operations that determines how it processes information or behaves.
- GPT-2includes · GPT-2 is a family of autoregressive, decoder-only Transformer language models developed by OpenAI that generate text by predicting successive tokens from preceding context.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.