Transformer architecture
A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Learn Generative AI with PyTorch
The internal design and structural organization of a Transformer model built around decoder blocks or self-attention components.
- How it is used
- Uses Transformer architectures for sequence prediction, especially translation and text generation.
- How deep it goes
- Works through encoder and decoder layers, attention calculations, tensor shapes, and a line-by-line implementation.
- Examples
- Uses English-to-French translation and a scaled-down GPT-style model to demonstrate architecture choices.
- What you can do after
- Builds an encoder-decoder Transformer from scratch and trains it for English-to-French translation.
Read how Learn Generative AI with PyTorch explains transformer architecture →
Natural Language Processing in Action, Second Edition
A transformer is a neural network architecture that uses attention mechanisms to process sequences and model relationships among their elements.
- How it is used
- Uses transformers for language generation, summarization, and translation, emphasizing stackable sequence representations.
- How deep it goes
- Explains tokenization, attention, positional encoding, and layer shapes, then implements and evaluates an encoder–decoder translator.
- Examples
- Uses German–English translation data, generated sentences, attention visualizations, and a model shape sanity check.
- What you can do after
- Apply it via PyTorch to train and evaluate a translator and inspect decoder attention weights.
Read how Natural Language Processing in Action, Second Edition explains transformer architecture →
Sutskever's List
A Transformer is a neural network architecture based primarily on attention mechanisms for modeling relationships among sequence elements.
- How it is used
- Frames the architectural shift from recurrent translation models to attention-based translation and language models.
- How deep it goes
- Explains direct token connections and identifies major components, while pointing to an annotated implementation for line-by-line detail.
- Examples
- Uses a “cheese”–“mice” dependency, translation BLEU scores, GPT-2, and a GPipe-trained Transformer.
- What you can do after
- Evaluate/understand why direct self-attention improved translation and training, while distinguishing plausible language generation from demonstrated understanding.
Read how Sutskever's List explains transformer architecture →
Deep Learning with Python, Third Edition
A neural-network architecture based on attention mechanisms, used here as the core architecture for large language models.
- How it is used
- Uses attention-based sequence modeling for translation, text classification, and large language models.
- How deep it goes
- Builds encoder blocks and a translation model, then adds positional embeddings and compares training results.
- Examples
- Compares English-to-Spanish translation models with and without position embeddings and revisits IMDb sentiment classification with a pretrained Transformer.
- What you can do after
- Build it from scratch using Keras layers to assemble and train an encoder–decoder translation model.
Read how Deep Learning with Python, Third Edition explains transformer architecture →
Build AI Drug Discovery Pipelines
A transformer architecture is a neural-network architecture that models relationships among sequence elements using attention and position-wise transformations, potentially with encoder and decoder stacks.
- How it is used
- Uses transformers to represent protein sequences for structure prediction and other drug-discovery tasks.
- How deep it goes
- Explains encoder–decoder and encoder-only architectures, attention, and positional encoding, while focusing on one component of the wider folding pipeline.
- Examples
- Uses SimpleFold and ESM-2 for protein folding and antibody classification to illustrate downstream use of protein language models.
- What you can do after
- Evaluate and understand how sequence embeddings support protein folding predictions, including the importance of held-out targets and per-residue confidence.
Read how Build AI Drug Discovery Pipelines explains transformer architecture →
Timeless Algorithms, The Foundational Ideas
A transformer is a neural-network architecture that uses attention to model relationships among sequence elements.
- How it is used
- Uses relational sequence modeling to explain the architectural basis of modern language models.
- How deep it goes
- Explains attention calculations and stacked components, then distinguishes simplified examples from full model behavior.
- Examples
- Works through pronoun resolution, semantic association, and autoregressive generation using simplified attention calculations.
- What you can do after
- Evaluates how attention constructs context and why parallel training, sequential generation, and reliability limits matter.
Read how Timeless Algorithms, The Foundational Ideas explains transformer architecture →
Build a Text-to-Image Generator (from Scratch)
A transformer architecture is a neural-network architecture that uses attention mechanisms to process relationships among elements in a sequence or other structured input.
- How it is used
- Uses translation to establish an attention-based architecture before applying sequence prediction to text-to-image generation.
- How deep it goes
- Builds an encoder–decoder translator from embeddings through attention, masks, training, and prediction, then extends the pipeline to image tokens.
- Examples
- Works through German-to-English translation and image generation from a panda prompt, including intermediate image-token outputs.
- What you can do after
- Build it from scratch in PyTorch as a trained encoder–decoder translator, then build a transformer-based text-to-image generator.
Read how Build a Text-to-Image Generator (from Scratch) explains transformer architecture →
Deep Learning with R, Third Edition
A Transformer is a neural-network architecture designed to process sets or sequences using attention-based contextual representations.
- How it is used
- Uses attention-based sequence modeling for translation, text classification, language modeling, and multimodal inputs.
- How deep it goes
- Develops encoder and decoder components, masking, positional embeddings, and a trained translation model, with comparisons to recurrent models.
- Examples
- Uses English-to-Spanish translation, IMDb sentiment classification, and image-question prompts to illustrate Transformer applications.
- What you can do after
- Builds a translation Transformer with R and Keras, then trains it and generates translations.
Read how Deep Learning with R, Third Edition explains transformer architecture →
Applied Reinforcement Learning
A neural-network architecture built around self-attention, position-wise feedforward transformations, residual connections, and normalization.
- How it is used
- Uses transformers to represent relationships among cooling zones for coordinated policy decisions and to process tokenized text in language models.
- How deep it goes
- Explains encoder sub-layers, residual connections, normalization, and stacking, and provides a PyTorch actor-critic implementation.
- Examples
- Illustrates an eight-zone data center cooling controller and next-token prediction from tokenized text.
- What you can do after
- Apply it via PyTorch encoder layers in a transformer-based actor-critic for coordinated cooling.
Read how Applied Reinforcement Learning explains transformer architecture →
Fight Fraud with Machine Learning
A transformer is a neural-network architecture that models relationships in sequences primarily through attention mechanisms rather than recurrence.
- How it is used
- Applies attention-based models to phishing, document-processing, and fake-voice fraud detection.
- How deep it goes
- Explains self-attention coefficients, multi-head concatenation, and encoder–decoder translation, then outlines Torch training without detailing layer internals.
- Examples
- Uses English-to-French translation, GPT phishing detection, KYC identity documents, and real-versus-fake voice classification.
- What you can do after
- Build it from scratch as a Torch classifier by defining layers and forward pass, then training, validating, and testing it.
Read how Fight Fraud with Machine Learning explains transformer architecture →
How Large Language Models Work
A transformer is a neural-network architecture that uses attention mechanisms to process relationships among elements in a sequence.
- How it is used
- Explains how language models turn token sequences into predictions through embeddings, attention, repeated computation, and sampling.
- How deep it goes
- Explains layers, attention, prediction types, and scaling behavior, but stops short of building an LLM from scratch or exposing proprietary details.
- Examples
- Uses Python dictionaries, preceding words, next-token prediction, translation, summarization, and a component diagram.
- What you can do after
- Evaluate/understand transformer outputs, model variants, sampling behavior, and practical deployment trade-offs.
Read how How Large Language Models Work explains transformer architecture →
Build an Advanced RAG Application (From Scratch)
The Transformer architecture is a neural-network architecture based on attention mechanisms for processing sequences and modeling relationships among their elements.
- How it is used
- Supports language understanding and generation tasks such as semantic search, translation, summarization, classification, and text generation.
- How deep it goes
- Explains embeddings, positional encoding, self-attention, multi-head attention, and encoder-decoder roles without emphasizing complex equations.
- Examples
- Contrasts processing “The quick brown fox jumps over the lazy dog” with RNNs, GRUs, and a Transformer.
- What you can do after
- Evaluate/understand how attention-based encoder and decoder components capture context and serve different language tasks.
Read how Build an Advanced RAG Application (From Scratch) explains transformer architecture →
Machines that Think
A neural-network architecture that processes sequence elements through attention-based interactions rather than requiring strictly sequential recurrence.
- How it is used
- Frames attention-based sequence processing as the architectural foundation for scalable language models and multimodal integration.
- How deep it goes
- Explains self-attention projections, block components, and encoder–decoder masking, with a high-level account of scaling and multimodal fusion.
- Examples
- Illustrates attention with “The sun rises in the east” and next-token generation with a masked decoder.
- What you can do after
- Evaluate and understand how attention, masking, and parallel processing shape Transformer capabilities and scaling constraints.
Read how Machines that Think explains transformer architecture →
AI-Powered Search
A transformer is a neural-network architecture that models relationships among sequence elements using attention and can encode and decode representations.
- How it is used
- Uses transformer embeddings to support semantic search and natural-language autocomplete.
- How deep it goes
- Explains contextual token representations and encoder–decoder separation, then outlines an embedding-based retrieval pipeline.
- Examples
- Illustrates contextual meaning with capital versus capitol and cross-language embeddings with English and Armenian greetings.
- What you can do after
- Evaluate and understand transformers for semantic search and question answering.
Read how AI-Powered Search explains transformer architecture →
Transformers in Action
A transformer is a neural network architecture that processes sequential and other structured data using attention mechanisms.
- How it is used
- Motivates attention-based sequence processing as a foundation for language models and other structured-data applications.
- How deep it goes
- Explains encoder–decoder flow, attention, feed-forward layers, residual connections, normalization, and positional encoding at a component level.
- Examples
- Illustrates English-to-French translation, sentiment in “The movie was not bad,” and gradient comparisons across sequence lengths.
- What you can do after
- Evaluate and understand transformer variants and their computational trade-offs for language tasks.
Read how Transformers in Action explains transformer architecture →
Transfer Learning for Natural Language Processing
A transformer is a neural network architecture that models sequences using attention mechanisms rather than recurrent connections.
- How it is used
- Motivates attention-based sequence modeling for translation and transfer learning across NLP tasks.
- How deep it goes
- Explains self-attention, encoder-decoder components, and parallelization; defers a complete BERT architectural discussion in the baseline treatment.
- Examples
- Uses machine translation, a sentence about cells and a cell phone, and a BERT classifier to illustrate transformer applications.
- What you can do after
- Evaluate and understand transformer components and their parallelization advantages, and apply pretrained BERT through Keras and TensorFlow Hub.
Read how Transfer Learning for Natural Language Processing explains transformer architecture →
Deep Learning with PyTorch, Second Edition
A neural network architecture for sequence modeling and content understanding based on attention rather than recurrence or convolution.
- How it is used
- Introduces attention-based sequence modeling for generation, translation, and content understanding.
- How deep it goes
- Builds a decoder in PyTorch and explains encoder-decoder information flow, while assigning full encoder-decoder implementation as an exercise.
- Examples
- Uses generated names, machine translation, and image classification to illustrate transformer applications.
- What you can do after
- Builds a decoder from scratch using PyTorch modules and its scaled dot product attention function.
Read how Deep Learning with PyTorch, Second Edition explains transformer architecture →
LLMs in Production
A transformer architecture is a neural-network architecture that models sequence relationships primarily through attention rather than recurrence or convolution.
- How it is used
- Positions transformer variants as the foundation for large language models and encoder-decoder transformers as architectures for conditional generation.
- How deep it goes
- Explains encoder-to-decoder data flow and training trade-offs, without detailing the attention computation.
- Examples
- Uses translation, summarization, T5, and a Llama 2 VRAM estimate to illustrate architecture and training.
- What you can do after
- Evaluate transformer architecture choices against task requirements, parallelism, training data, and VRAM constraints.
Read how LLMs in Production explains transformer architecture →
Grokking AI Applications
A transformer is a neural-network architecture for sequential data that uses attention to relate elements of an input sequence.
- How it is used
- Explains how GPT-style systems process ordered text and generate responses through attention, deep neural networks, and token prediction.
- How deep it goes
- Defines attention and layered processing with an example, but stops short of mathematical or implementation-level internals.
- Examples
- Uses GPT-3, the question about Italy's capital, token-by-token generation, and diagrams of GPT architecture and training.
- What you can do after
- Understand how attention and layered neural networks support GPT-style sequence processing and text generation.
Read how Grokking AI Applications explains transformer architecture →
The AI Pocket Book
A neural network architecture that contextualizes each token using surrounding tokens rather than compressing the whole prompt into one summary vector.
- How it is used
- Explains how current LLMs turn token embeddings into context-sensitive representations for next-token prediction.
- How deep it goes
- Defines contextualization and attention-based updates, but stops short of exact calculations and implementation details.
- Examples
- Uses dog-versus-tree “bark” prompts, token embeddings, attention scores, and a compact GPT-2 implementation as concrete illustrations.
- What you can do after
- Evaluate/understand why transformers preserve contextual information better than single-summary LSTM representations.
Read how The AI Pocket Book explains transformer architecture →
Time Series Forecasting Using Foundation Models
A neural-network architecture built from stacked attention and feed-forward processing components, typically organized into encoder and decoder parts.
- How it is used
- Uses attention-based sequence processing to explain and support time-series foundation-model forecasting.
- How deep it goes
- Explains embeddings, positional encoding, encoder-decoder flow, attention variants, masking, and projection, but defers attention mathematics.
- Examples
- Uses electricity-demand forecasting with temperature, plus trend and seasonal patterns as attention-learning examples.
- What you can do after
- Evaluate transformer-based forecasters and diagnose architectural limitations involving exogenous features, multiple series, and fine-tuning.
Read how Time Series Forecasting Using Foundation Models explains transformer architecture →
Building Specialized AI Systems
A neural-network architecture based primarily on attention mechanisms for processing sequences.
- How it is used
- Explains the backbone used to build contextual language representations before adapting them through fine-tuning.
- How deep it goes
- Explains inputs, encoder-decoder components, attention computation, and architectural motivation, but stops short of implementation code.
- Examples
- Uses language translation, tokenized subword inputs, attention weights, and encoder-decoder architecture diagrams.
- What you can do after
- Evaluate how attention, masking, positional information, and encoder-decoder pathways support sequence modeling and later fine-tuning.
Read how Building Specialized AI Systems explains transformer architecture →
Machine Learning Algorithms in Depth
A transformer model is a neural network architecture that uses attention-based mechanisms to model contextual relationships in structured sequences or other data.
- How it is used
- Uses transformer architecture to connect attention mechanisms with an implemented text-classification task.
- How deep it goes
- Explains query–key–value weighting and major architectural components, then provides a classifier implementation using Keras/TensorFlow layers.
- Examples
- Uses a translation attention heatmap, a transformer architecture diagram, and text-classifier training and validation plots.
- What you can do after
- Apply it via Keras/TensorFlow to text classification and inspect training and validation loss and accuracy.
Read how Machine Learning Algorithms in Depth explains transformer architecture →
Inside Deep Learning: Math, Algorithms, Models
A sequence model built from positional encoding and multiheaded attention, used here as the standard original design for NLP and other tasks.
- How it is used
- Uses transformers as an alternative to RNNs for sequence tasks, especially when depth, data, and compute can scale.
- How deep it goes
- Explains encoder and decoder blocks, residual connections, and autoregressive masking, and includes a classifier implementation.
- Examples
- Uses sentiment classification, machine translation, autoregressive prediction, and a PyTorch transformer classifier as examples.
- What you can do after
- Evaluate when transformer models suit sequence tasks and understand their block structure, masking needs, and compute tradeoffs.
Read how Inside Deep Learning: Math, Algorithms, Models explains transformer architecture →
Grokking AI Algorithms, Second Edition
The Transformer is a neural-network architecture based primarily on attention mechanisms for processing sequences and modeling context.
- How it is used
- Uses attention-based sequence processing to power language models that generate text, translate languages, answer questions, and write code.
- How deep it goes
- Explains self-attention, block components, architecture alternatives, and hyperparameters, but stops short of equations or implementation details.
- Examples
- Uses a toy language model with 34 tokens, 32-token context, two layers, one attention head, and small feed-forward networks.
- What you can do after
- Evaluate Transformer designs by comparing vocabulary, context, width, depth, attention-head, and feed-forward-size choices.
Read how Grokking AI Algorithms, Second Edition explains transformer architecture →
Understanding AI
A Transformer is a neural-network architecture that uses attention mechanisms to model relationships among sequence elements, often in parallel.
- How it is used
- Frames modern AI capabilities as attention-based pattern prediction that can generalize across many tasks.
- How deep it goes
- Explains attention, parallel sequence processing, scalable training, and broad task generalization without detailing internal computations.
- Examples
- Uses distant relationships in text and multimodal actions such as image questions, screenshot debugging, and voice-text interaction.
- What you can do after
- Evaluate/understand why attention-based models scale beyond narrow tasks while still relying on statistical patterns rather than human understanding.
Read how Understanding AI explains transformer architecture →
Hugging Face in Action
The Transformer architecture is a neural-network architecture based on attention mechanisms for processing sequences and other structured inputs.
- How it is used
- Deploys Transformer models through high-level pipelines for practical NLP tasks such as classification, generation, and summarization.
- How deep it goes
- Explains self-attention, embeddings, positions, and encoder-decoder roles briefly, while stopping short of detailed Transformer internals.
- Examples
- Demonstrates IMDb sentiment analysis with a pretrained DistilBERT model and Hugging Face’s pipeline API.
- What you can do after
- Apply it via Hugging Face pipelines to run pretrained models for classification and other NLP workflows.
Read how Hugging Face in Action explains transformer architecture →
Domain-Specific Small Language Models
A neural network architecture for sequence modeling built around self-attention and parallel processing, underlying both large and small language models.
- How it is used
- Frames sequence modeling around processing whole text inputs in parallel rather than recurrent word-by-word steps.
- How deep it goes
- Defines self-attention and explains its parallelism advantage over recurrent networks, while leaving multimodal applications out of scope.
- Examples
- Contrasts sequential RNN processing with whole-sequence Transformers and uses GPT-Neo large for content-generation inference.
- What you can do after
- Apply it via Hugging Face Transformers and Accelerate to run generation while considering speed, accuracy, and compute cost.
Read how Domain-Specific Small Language Models explains transformer architecture →
Introduction to Generative AI, Second Edition
A transformer is a neural-network architecture for sequence modeling that uses attention, especially self-attention, rather than recurrence.
- How it is used
- Explains the architectural shift that enabled modern language models and generative AI applications.
- How deep it goes
- Defines attention and self-attention through intuition and history, but stops short of implementation details, mathematics, or training procedures.
- Examples
- Uses a cat-and-mouse sentence, translation benchmarks, GPT, and attention visualizations to illustrate the architecture.
- What you can do after
- Evaluate/understand it by relating parallel self-attention to long-range context, scalable training, and LLM applications.
Read how Introduction to Generative AI, Second Edition explains transformer architecture →
Build AI-Enhanced Web Apps
A transformer model is a neural-network architecture that uses attention mechanisms to model contextual relationships in sequence data.
- How it is used
- Supports generative AI and language-model applications requiring global context, contextual relationships, and large-scale sequence processing.
- How deep it goes
- Defines self-attention, parallel processing, and contextual relationships with an example, but stops short of implementation details.
- Examples
- Uses pronoun resolution in a sentence to demonstrate how attention captures relationships separated by multiple words.
- What you can do after
- Evaluate transformer suitability for applications involving global context, language modeling, and sequence relationships.
Read how Build AI-Enhanced Web Apps explains transformer architecture →
Retrieval Augmented Generation, The Foundational Ideas
A transformer model is a neural-network architecture based on attention mechanisms for processing sequences and modeling contextual relationships.
- How it is used
- Supports contextual encoding for semantic search and similarity-based routing.
- How deep it goes
- Defines parallel processing and self-attention with contextual examples, then stops short of implementation details.
- Examples
- Uses bank disambiguation and customer-support ticket matching as concrete examples.
- What you can do after
- Evaluate contextual representations and recognize pairwise inference costs in large-scale similarity search.
Read how Retrieval Augmented Generation, The Foundational Ideas explains transformer architecture →
Rearchitecting LLMs
A transformer is a neural-network architecture that processes representations using attention and position-wise feed-forward components.
- How it is used
- Maps model components to memory, compute, and re-architecture decisions for inference optimization.
- How deep it goes
- Inspects model structure, constructor, attention, MLP, and activations; leaves full inference-time dynamics for later.
- Examples
- Uses DistilGPT2's printed module tree and fictional Llama-3.2 activation patterns.
- What you can do after
- Understand how model components relate to optimization choices and how activations reveal input-dependent behavior.
Read how Rearchitecting LLMs explains transformer architecture →
Build a Large Language Model (From Scratch)
a neural network architecture based on self-attention mechanisms that enables processing of sequential data and underpins many large language models.
- How it is used
- Organizes language processing into contextual encoding and sequential generation for translation and large language models.
- How deep it goes
- Defines encoder, decoder, and self-attention, but defers self-attention implementation and input preprocessing to later treatment.
- Examples
- Uses English-to-German translation, including generating “Beispiel” after “Das ist ein.”
- What you can do after
- Evaluate/understand how encoder-decoder flow and self-attention support translation and GPT-like text generation.
Read how Build a Large Language Model (From Scratch) explains transformer architecture →
Build a Website with ChatGPT
A transformer is a neural-network architecture that processes sequences using attention mechanisms to model relationships among their components.
- How it is used
- Explains why GPT can turn plain-English website requests into relevant generated code.
- How deep it goes
- Defines transformer as prioritizing important request components but stops short of describing attention layers or implementation.
- Examples
- Uses web-page requests and the prompt-copy-save-deploy cycle as practical examples.
- What you can do after
- Apply it via ChatGPT by writing prompts, copying generated code, saving an HTML file, and deploying it.
Read how Build a Website with ChatGPT explains transformer architecture →
Data Analysis with LLMs
The Transformer is a neural-network architecture designed to model relationships in sequences, especially sequences with variable-length inputs or outputs.
- How it is used
- Supports learning tasks with variable-length inputs or outputs, especially text, and underlies generative AI.
- How deep it goes
- Defines its sequence-learning role and broad prevalence without describing internal components or implementation.
- Examples
- Illustrates variable-length sequences with text documents and names document summarization and code writing as GPT tasks.
Read how Data Analysis with LLMs explains transformer architecture →
Generative AI in Action
The Transformer architecture is a neural-network architecture based primarily on attention mechanisms for processing sequences.
- How it is used
- Enables foundational models to understand and generate language for summarization, translation, classification, question answering, and content generation.
- How deep it goes
- Explains self-attention, contextual dependencies, parallel computation, and next-item prediction, but stops short of mathematical or layer-level internals.
- Examples
- Uses the completion “The cat sat on the...” to illustrate attention-weighted prediction of “mat,” alongside GPT and BERT as Transformer models.
- What you can do after
- Evaluate Transformer use for language generation while recognizing its substantial computational demands for training and use.
Read how Generative AI in Action explains transformer architecture →
Coding with AI
A Transformer is a neural-network architecture that processes sequence elements using attention and can process them in parallel.
- How it is used
- Underpins large language models for coding tools, chatbots, translation, and content generation.
- How deep it goes
- Explains input identifiers, positions, self-attention, parallel processing, and scaling, without detailing layer internals.
- Examples
- Uses “Jane helps Joe,” “The dog chased the cat,” and a figure showing positions and relationships.
- What you can do after
- Evaluate/understand it by tracing how prompts become context-sensitive representations for language applications.
Optimization Algorithms: AI techniques for design, planning, and control problems
A transformer architecture is a neural-network architecture that processes sequences primarily through attention mechanisms rather than recurrence.
- How it is used
- Processes sequential data such as text and translation by prioritizing relevant input parts during prediction.
- How deep it goes
- Defines attention intuitively as relevance scores but does not detail Transformer layers, equations, or training.
- Examples
- Uses word-by-word reading, long sentences, translation, and attention coefficients as explanatory examples.
- What you can do after
- Understand how attention can reduce context loss and confusion in long or complex sequences.
Essential GraphRAG
A transformer architecture is a neural-network architecture that processes sequences using attention-based representations and supports efficient language modeling.
- How it is used
- Explains the architecture underlying LLM text processing, generation, and instruction-following behavior.
- How deep it goes
- Defines the architecture’s role and training objective but stops short of explaining internal mechanisms or implementation details.
- Examples
- Uses next-word completion, “Never gonna” to “give you up,” and haiku generation as concrete LLM examples.
- What you can do after
- Evaluate/understand how transformer-based LLMs generate text and why external retrieval can improve factual responses.
Read how Essential GraphRAG explains transformer architecture →
CUDA for LLMs
A transformer is a neural-network architecture that processes sequences using attention and position-aware transformations.
- How it is used
- Frames transformer architecture as the target for CUDA inference-kernel optimization.
- How deep it goes
- Defines sequence processing through attention and position-aware transformations, and establishes terminology and a numerical baseline.
- Examples
- Uses a PyTorch baseline as the numerical reference for inference kernels.
- What you can do after
- Evaluate CUDA inference kernels against a numerically matching PyTorch baseline.
Also covered in
- Data Without Labels
- Causal AI
- AI Agents in Action, Second Edition
- AI Agents in Action
- Data Storytelling with Altair and AI
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
- Investing for Programmers
- AI-Powered Developer: Build software with ChatGPT and Copilot
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- natural language processingNatural language processing is the field of computing concerned with computational methods for analyzing, interpreting, transforming, and generating human language in text or speech.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
Related concepts
- attentionuses · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- decoding strategyhas part · A decoding strategy is a procedure for converting an encoded or internal representation into a target output, including selecting tokens during sequence generation.
- encoderhas part · An encoder is a component or process that transforms input into an encoded or internal representation for subsequent processing, transmission, or decoding.
- Vision Transformerincludes · A Vision Transformer is a Transformer-based neural-network architecture that processes images as sequences of tokenized regions using attention.
- BERTincludes · BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- model architecturekind of · Model architecture is the structural arrangement of a computational model’s components, connections, and operations that determines how it processes inputs and produces outputs.
Questions about transformer architecture
What is transformer architecture?
A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Which Manning books explain transformer architecture?
Learn Generative AI with PyTorch, Natural Language Processing in Action, Second Edition, Sutskever's List, Deep Learning with Python, Third Edition, Build AI Drug Discovery Pipelines, Timeless Algorithms, The Foundational Ideas, Build a Text-to-Image Generator (from Scratch), Deep Learning with R, Third Edition and 32 more.
What should I know before learning transformer architecture?
Manning books that teach transformer architecture usually introduce language model, recurrent neural network, natural language processing and attention first.
What kinds of transformer architecture are there?
Kinds of transformer architecture include Vision Transformer and BERT.
What is transformer architecture a kind of?
Transformer architecture is a kind of model architecture. Model architecture is the structural arrangement of a computational model’s components, connections, and operations that determines how it processes inputs and produces outputs.
What are the parts of transformer architecture?
Transformer architecture includes decoding strategy and encoder.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.







































