Encoder-only transformer

As explained in Effective Conversational AI

An encoder-only architecture is a language-model architecture that transforms input text into contextual representations without generating output sequences.

An encoder-only transformer turns input text into contextual representations rather than generating an output sequence. It is suited to non-generative tasks where understanding the input matters more than producing new text. section 4.1.4

Use an encoder-only transformer when you need to:

• Train predictive models from text embeddings. section 4.1.4
• Extract meaningful context from an input instead of predicting the next token. section 4.1.4
• Fine-tune with labeled data for a defined prediction task. section 4.1.4

Figure 4.4 Traditional classification models use supervised learning to predict one of several predefined classifications. They look for the intent, or meaning, of a user input. Generative models use decoding transformers to create a text completion. They predict the next sequence of tokens (loosely, words or characters) that are most likely to occur after the user input.

Compare encoder-oriented classification with decoder-oriented generation: one predicts among predefined classifications, while the other creates a text completion. section 4.1.4

How Effective Conversational AI covers it

How deep it goes

Explains its task orientation and fine-tuning requirement, but stops short of implementation details.

Examples

Uses predictive models based on text embeddings as its primary application example.

What it assumes

Assumes labeled data for fine-tuning while focusing on extracting input context rather than generating language.

What you can do after

Evaluate model fit for predictive language tasks that do not require generated output sequences.

How it connects

Contrasts with decoder-only architectures, encoder-decoder architectures, classification models, generative AI, and text embeddings.

liveBookmanning.com

Read Effective Conversational AI on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)An encoder-only transformer consists of encoder layers that convert an input sequence into continuous vector representations.
• 

  liveBookmanning.com
  Transformers in ActionAn encoder-only architecture is a sequence-model architecture that uses an encoder without a separate output decoder.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsAn encoder-only transformer is a transformer architecture composed of encoder-style sequence-processing layers without a separate decoder.

Compare how 4 books explain encoder-only transformer →

Related concepts

• BERTincludes · BERT is a pretrained Transformer encoder language model that uses bidirectional context to produce contextual representations of text.
• transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• decoder-only transformercontrasts with · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
• Moiraiused by · Moirai is a foundation model designed for time-series forecasting across varied datasets.
• masked language modelinguses · Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden or replaced in an input sequence using surrounding context.
• Transformer encoder layerhas part · A Transformer encoder layer is a sequence-processing module that combines self-attention and position-wise feed-forward processing, typically with residual connections and normalization.

This explainer was generated by Manning's knowledge graph from the text of Effective Conversational AI. The section links point to the parts of the book it draws on.