Encoder-only transformer
An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
Build a Text-to-Image Generator (from Scratch)
An encoder-only transformer consists of encoder layers that convert an input sequence into continuous vector representations.
- How it is used
- Supports input understanding and task-specific prediction, including image classification through a Vision Transformer.
- How deep it goes
- Explains token preparation, stacked blocks, normalization, residuals, MLP dimensions, and a concrete four-layer implementation.
- Examples
- Uses CIFAR-10 images, 4 × 4 image patches, a class token, and PyTorch-style MLP, block, and encoder implementations.
- What you can do after
- Build it from scratch for image classification by assembling embeddings, positional encodings, encoder blocks, and a training objective.
Read how Build a Text-to-Image Generator (from Scratch) explains encoder-only transformer →
Transformers in Action
An encoder-only architecture is a sequence-model architecture that uses an encoder without a separate output decoder.
- How it is used
- Serves classification, text understanding, and retrieval workloads that do not require full generative capability.
- How deep it goes
- Explains bidirectional attention, single-pass inference, and masked pretraining, with abstracted encoder and prediction-head listings rather than a complete training pipeline.
- Examples
- Uses BERT, RoBERTa, ModernBERT, a Fairseq encoder listing, and a masked “Paris” prediction.
- What you can do after
- Evaluate/understand it as an efficient choice for classification or retrieval instead of autoregressive generation.
Read how Transformers in Action explains encoder-only transformer →
Time Series Forecasting Using Foundation Models
An encoder-only transformer is a transformer architecture composed of encoder-style sequence-processing layers without a separate decoder.
- How it is used
- Deploys encoder-style sequence processing for probabilistic time-series forecasting with multivariate and exogenous inputs.
- How deep it goes
- Explains patching, frequency selection, multivariate flattening, and mixture outputs, but does not alter the encoder internals.
- Examples
- Demonstrates frequency-specific patch lengths and diagrams the patched sequence, flattened variates, and probabilistic output.
- What you can do after
- Apply it via Moirai's Python package for probabilistic forecasting, while accounting for GluonTS and hardware requirements.
Read how Time Series Forecasting Using Foundation Models explains encoder-only transformer →
Effective Conversational AI
An encoder-only architecture is a language-model architecture that transforms input text into contextual representations without generating output sequences.
- How it is used
- Supports non-generative language tasks by extracting contextual meaning for predictive models and text embeddings.
- How deep it goes
- Explains its task orientation and fine-tuning requirement, but stops short of implementation details.
- Examples
- Uses predictive models based on text embeddings as its primary application example.
- What you can do after
- Evaluate model fit for predictive language tasks that do not require generated output sequences.
Read how Effective Conversational AI explains encoder-only transformer →
Related concepts
- BERTincludes · BERT is a pretrained Transformer encoder language model that uses bidirectional context to produce contextual representations of text.
- transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- decoder-only transformercontrasts with · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
- Moiraiused by · Moirai is a foundation model designed for time-series forecasting across varied datasets.
- masked language modelinguses · Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden or replaced in an input sequence using surrounding context.
- Transformer encoder layerhas part · A Transformer encoder layer is a sequence-processing module that combines self-attention and position-wise feed-forward processing, typically with residual connections and normalization.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.