Encoder-only transformer
An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
Build AI Drug Discovery Pipelines
An encoder-only transformer is a transformer consisting of encoder layers that produces contextual representations without an autoregressive decoder.
- How it is used
- Uses encoder-only transformers to represent protein sequences for folding and property prediction.
- How deep it goes
- Explains parallel encoding and attention components, contrasting the encoder-only architecture with recurrent and encoder-decoder models.
- Examples
- Uses ESM-2 and protein sequences to illustrate residue-level representations.
- What you can do after
- Evaluate/understand when an encoder-only architecture is appropriate instead of an encoder-decoder model.
Read how Build AI Drug Discovery Pipelines explains encoder-only transformer →
Build a Text-to-Image Generator (from Scratch)
An encoder-only transformer consists of encoder layers that convert an input sequence into continuous vector representations.
- How it is used
- Supports input understanding and task-specific prediction, including image classification through a Vision Transformer.
- How deep it goes
- Explains token preparation, stacked blocks, normalization, residuals, MLP dimensions, and a concrete four-layer implementation.
- Examples
- Uses CIFAR-10 images, 4 × 4 image patches, a class token, and PyTorch-style MLP, block, and encoder implementations.
- What you can do after
- Build it from scratch for image classification by assembling embeddings, positional encodings, encoder blocks, and a training objective.
Read how Build a Text-to-Image Generator (from Scratch) explains encoder-only transformer →
Transformers in Action
An encoder-only architecture is a sequence-model architecture that uses an encoder without a separate output decoder.
- How it is used
- Positions encoder-only models as efficient choices for text understanding, classification, embeddings, and retrieval rather than open-ended generation.
- How deep it goes
- Explains bidirectional attention and masked pretraining, with an abstracted encoder and a masked-token projection listing.
- Examples
- Uses BERT, RoBERTa, ModernBERT, a masked capital-city sentence, and Prompt Guard 2 to illustrate models and applications.
- What you can do after
- Evaluate encoder-only models for classification, retrieval, and prompt filtering against the need for generative capability and inference efficiency.
Read how Transformers in Action explains encoder-only transformer →
Time Series Forecasting Using Foundation Models
An encoder-only transformer is a transformer architecture composed of encoder-style sequence-processing layers without a separate decoder.
- How it is used
- Deploys encoder-style sequence processing for probabilistic time-series forecasting with multivariate and exogenous inputs.
- How deep it goes
- Explains patching, frequency selection, multivariate flattening, and mixture outputs, but does not alter the encoder internals.
- Examples
- Demonstrates frequency-specific patch lengths and diagrams the patched sequence, flattened variates, and probabilistic output.
- What you can do after
- Apply it via Moirai's Python package for probabilistic forecasting, while accounting for GluonTS and hardware requirements.
Read how Time Series Forecasting Using Foundation Models explains encoder-only transformer →
Building Specialized AI Systems
An encoder-only model consists of encoder layers that can represent each token using context from both directions.
- How it is used
- Supports context-rich representation learning and classification rather than step-by-step text generation.
- How deep it goes
- Explains bidirectional attention, pretraining objectives, architecture choice, and long-context cost, without detailing a specific implementation.
- Examples
- Illustrates the layout with an encoder–decoder Transformer diagram showing self-attention, feed-forward blocks, residual connections, and normalization.
- What you can do after
- Evaluate whether bidirectional representations fit classification and representation tasks, while recognizing long-context attention costs.
Read how Building Specialized AI Systems explains encoder-only transformer →
LLM Evaluation and Alignment, The Foundational Ideas
An encoder-only Transformer is a Transformer model composed of bidirectional encoder layers for representing an entire input sequence.
- How it is used
- Compares complete texts for semantic similarity and evaluation rather than generating text.
- How deep it goes
- Explains bidirectional context and embedding extraction, then stops short of detailing Transformer layer internals.
- Examples
- Uses legal-context interpretation, paraphrased committee passages, [CLS] pooling, and mean pooling.
- What you can do after
- Builds text embeddings with BERT and compares them using cosine similarity.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains encoder-only transformer →
Effective Conversational AI
An encoder-only architecture is a language-model architecture that transforms input text into contextual representations without generating output sequences.
- How it is used
- Supports non-generative language tasks by extracting contextual meaning for predictive models and text embeddings.
- How deep it goes
- Explains its task orientation and fine-tuning requirement, but stops short of implementation details.
- Examples
- Uses predictive models based on text embeddings as its primary application example.
- What you can do after
- Evaluate model fit for predictive language tasks that do not require generated output sequences.
Read how Effective Conversational AI explains encoder-only transformer →
Also covered in
- How Large Language Models Work
- LLMs in Production
- Learn Generative AI with PyTorch
- Machines that ThinkIn early access (MEAP)
- Build an Advanced RAG Application (From Scratch)In early access (MEAP)
What to know first
- BERTBERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- classifierA classifier is a predictive model that assigns an input to one or more discrete classes or categories.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Related concepts
- BERTincludes · BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- transformer architecturekind of · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- decoder-only transformercontrasts with · A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context.
- Transformer encoder layerhas part · A Transformer encoder layer is a sequence-processing module that uses self-attention and feed-forward transformations to produce contextualized representations.
- masked language modelinguses · Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context.
- Moiraiincludes · Moirai is a foundation model designed for time-series forecasting across varied datasets.
Questions about encoder-only transformer
What is encoder-only transformer?
An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
Which Manning books explain encoder-only transformer?
Build AI Drug Discovery Pipelines, Build a Text-to-Image Generator (from Scratch), Transformers in Action, Time Series Forecasting Using Foundation Models, Building Specialized AI Systems, LLM Evaluation and Alignment, The Foundational Ideas and Effective Conversational AI.
What should I know before learning encoder-only transformer?
Manning books that teach encoder-only transformer usually introduce BERT, classifier, token and transformer architecture first.
What is the difference between encoder-only transformer and decoder-only transformer?
An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder. A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context.
What kinds of encoder-only transformer are there?
Kinds of encoder-only transformer include BERT and Moirai.
What is encoder-only transformer a kind of?
Encoder-only transformer is a kind of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






