Encoder-only transformer

As explained in Time Series Forecasting Using Foundation Models

An encoder-only transformer is a transformer architecture composed of encoder-style sequence-processing layers without a separate decoder.

An encoder-only transformer processes a sequence with encoder-style layers and no separate decoder. In Moirai, it receives embedded time-series patches for probabilistic forecasting. section 6.1.1 section 6.6

Its forecasting implementation combines three main choices:

• Patching groups data points into tokens before embedding, reducing token count while preserving long input windows. section 6.1.1
• Frequency-dependent patch lengths are selected automatically rather than chosen manually. section 6.1.1
• Flattening multivariate inputs into one long sequence lets the model handle an arbitrary number of variates. section 6.1.1
• A mixture distribution produces a flexible distribution of future values instead of only a single forecast. section 6.1.1

Frequency-dependent patching

Yearly and quarterly data use patches of length 8; monthly data use lengths 8, 16, and 32; hourly data use 32 and 64. Each patch becomes an embedded token. section 6.1.1

Figure 6.1 Architecture of Moirai. Here, the series is patched before entering the encoder-only transformer. Then, a mixture of distributions generates a flexible output distribution of future values.

Trace the path from patched input series through the encoder-only transformer to mixture-distribution forecasts. section 6.1.1

How Time Series Forecasting Using Foundation Models covers it

How deep it goes

Explains patching, frequency selection, multivariate flattening, and mixture outputs, but does not alter the encoder internals.

Examples

Demonstrates frequency-specific patch lengths and diagrams the patched sequence, flattened variates, and probabilistic output.

What you can do after

Apply it via Moirai's Python package for probabilistic forecasting, while accounting for GluonTS and hardware requirements.

How it connects

Connects patching, embedding layers, multivariate data, exogenous features, and mixture distributions.

liveBookmanning.com

Read Time Series Forecasting Using Foundation Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)An encoder-only transformer consists of encoder layers that convert an input sequence into continuous vector representations.
• 

  liveBookmanning.com
  Transformers in ActionAn encoder-only architecture is a sequence-model architecture that uses an encoder without a separate output decoder.
• 

  liveBookmanning.com
  Effective Conversational AIAn encoder-only architecture is a language-model architecture that transforms input text into contextual representations without generating output sequences.

Compare how 4 books explain encoder-only transformer →

Related concepts

• BERTincludes · BERT is a pretrained Transformer encoder language model that uses bidirectional context to produce contextual representations of text.
• transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• decoder-only transformercontrasts with · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
• Moiraiused by · Moirai is a foundation model designed for time-series forecasting across varied datasets.
• masked language modelinguses · Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden or replaced in an input sequence using surrounding context.
• Transformer encoder layerhas part · A Transformer encoder layer is a sequence-processing module that combines self-attention and position-wise feed-forward processing, typically with residual connections and normalization.

This explainer was generated by Manning's knowledge graph from the text of Time Series Forecasting Using Foundation Models. The section links point to the parts of the book it draws on.