liveBook · Manning

Encoder-only transformer

Also called encoder-only model, encoder-only architecture

An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.

How 7 Manning books explain it

Build AI Drug Discovery Pipelines

An encoder-only transformer is a transformer consisting of encoder layers that produces contextual representations without an autoregressive decoder.

How it is used
Uses encoder-only transformers to represent protein sequences for folding and property prediction.
How deep it goes
Explains parallel encoding and attention components, contrasting the encoder-only architecture with recurrent and encoder-decoder models.
Examples
Uses ESM-2 and protein sequences to illustrate residue-level representations.
What you can do after
Evaluate/understand when an encoder-only architecture is appropriate instead of an encoder-decoder model.

Read how Build AI Drug Discovery Pipelines explains encoder-only transformer →

Build a Text-to-Image Generator (from Scratch)

An encoder-only transformer consists of encoder layers that convert an input sequence into continuous vector representations.

How it is used
Supports input understanding and task-specific prediction, including image classification through a Vision Transformer.
How deep it goes
Explains token preparation, stacked blocks, normalization, residuals, MLP dimensions, and a concrete four-layer implementation.
Examples
Uses CIFAR-10 images, 4 × 4 image patches, a class token, and PyTorch-style MLP, block, and encoder implementations.
What you can do after
Build it from scratch for image classification by assembling embeddings, positional encodings, encoder blocks, and a training objective.

Read how Build a Text-to-Image Generator (from Scratch) explains encoder-only transformer →

Transformers in Action

An encoder-only architecture is a sequence-model architecture that uses an encoder without a separate output decoder.

How it is used
Positions encoder-only models as efficient choices for text understanding, classification, embeddings, and retrieval rather than open-ended generation.
How deep it goes
Explains bidirectional attention and masked pretraining, with an abstracted encoder and a masked-token projection listing.
Examples
Uses BERT, RoBERTa, ModernBERT, a masked capital-city sentence, and Prompt Guard 2 to illustrate models and applications.
What you can do after
Evaluate encoder-only models for classification, retrieval, and prompt filtering against the need for generative capability and inference efficiency.

Read how Transformers in Action explains encoder-only transformer →

Time Series Forecasting Using Foundation Models

An encoder-only transformer is a transformer architecture composed of encoder-style sequence-processing layers without a separate decoder.

How it is used
Deploys encoder-style sequence processing for probabilistic time-series forecasting with multivariate and exogenous inputs.
How deep it goes
Explains patching, frequency selection, multivariate flattening, and mixture outputs, but does not alter the encoder internals.
Examples
Demonstrates frequency-specific patch lengths and diagrams the patched sequence, flattened variates, and probabilistic output.
What you can do after
Apply it via Moirai's Python package for probabilistic forecasting, while accounting for GluonTS and hardware requirements.

Read how Time Series Forecasting Using Foundation Models explains encoder-only transformer →

Building Specialized AI Systems

An encoder-only model consists of encoder layers that can represent each token using context from both directions.

How it is used
Supports context-rich representation learning and classification rather than step-by-step text generation.
How deep it goes
Explains bidirectional attention, pretraining objectives, architecture choice, and long-context cost, without detailing a specific implementation.
Examples
Illustrates the layout with an encoder–decoder Transformer diagram showing self-attention, feed-forward blocks, residual connections, and normalization.
What you can do after
Evaluate whether bidirectional representations fit classification and representation tasks, while recognizing long-context attention costs.

Read how Building Specialized AI Systems explains encoder-only transformer →

LLM Evaluation and Alignment, The Foundational Ideas

An encoder-only Transformer is a Transformer model composed of bidirectional encoder layers for representing an entire input sequence.

How it is used
Compares complete texts for semantic similarity and evaluation rather than generating text.
How deep it goes
Explains bidirectional context and embedding extraction, then stops short of detailing Transformer layer internals.
Examples
Uses legal-context interpretation, paraphrased committee passages, [CLS] pooling, and mean pooling.
What you can do after
Builds text embeddings with BERT and compares them using cosine similarity.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains encoder-only transformer →

Effective Conversational AI

An encoder-only architecture is a language-model architecture that transforms input text into contextual representations without generating output sequences.

How it is used
Supports non-generative language tasks by extracting contextual meaning for predictive models and text embeddings.
How deep it goes
Explains its task orientation and fine-tuning requirement, but stops short of implementation details.
Examples
Uses predictive models based on text embeddings as its primary application example.
What you can do after
Evaluate model fit for predictive language tasks that do not require generated output sequences.

Read how Effective Conversational AI explains encoder-only transformer →

Also covered in

What to know first

Related concepts

Questions about encoder-only transformer

What is encoder-only transformer?

An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.

Which Manning books explain encoder-only transformer?

Build AI Drug Discovery Pipelines, Build a Text-to-Image Generator (from Scratch), Transformers in Action, Time Series Forecasting Using Foundation Models, Building Specialized AI Systems, LLM Evaluation and Alignment, The Foundational Ideas and Effective Conversational AI.

What should I know before learning encoder-only transformer?

Manning books that teach encoder-only transformer usually introduce BERT, classifier, token and transformer architecture first.

What is the difference between encoder-only transformer and decoder-only transformer?

An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder. A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context.

What kinds of encoder-only transformer are there?

Kinds of encoder-only transformer include BERT and Moirai.

What is encoder-only transformer a kind of?

Encoder-only transformer is a kind of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.