Moirai-MoE

As explained in Time Series Forecasting Using Foundation Models

Moirai-MoE is a Moirai-based universal time-series forecasting model that uses fixed-length, nonoverlapping patches across frequencies.

Moirai-MoE is a Moirai-based universal time-series forecaster that patches every frequency with the same fixed, nonoverlapping patch length. A decoder-only transformer generates forecast patches sequentially, while a sparse mixture-of-experts layer specializes in different time-series patterns. section 6.2 section 6.2.1 section 6.2.2

Its main implementation choices are:

• Uses one input projection for all frequencies instead of training frequency-specific embedding layers. section 6.2.1
• Routes tokens through a gating function and 32 feed-forward experts; at most two experts are activated simultaneously. section 6.2.2
• Generates one patch at a time, with each predicted patch informing the next prediction. section 6.2.2
• Provides small and base configurations with 117 million and 935 million parameters, respectively. section 6.2.3

Figure 6.5 Architecture of Moirai-MoE. Unlike the first version of Moirai, the new version uses a decoder-only transformer. Inside the decoder-only transformer, a gating function and a mixture-of-experts layer have been added. The gating function learns which expert to activate depending on the data, with a maximum of two experts being activated at the same time. The activation of an expert is indicated by a solid arrow.

Use the architecture diagram to trace patching, decoder self-attention, gating, sparse expert activation, and forecast projection. section 6.2

How Time Series Forecasting Using Foundation Models covers it

How deep it goes

Explains patching, sequential decoding, expert routing, parameter sizes, and sparse inference, but not training implementation details.

Examples

Uses frequency-independent patches, 32 pattern-specialized experts, and increasing- or decreasing-trend experts as concrete examples.

What you can do after

Evaluate how shared patching and sparse expert activation trade architectural generality against model size and inference cost.

How it connects

Connects patching, embeddings, decoder-only transformers, self-attention, gating functions, mixture of experts, and probabilistic forecasts.

liveBookmanning.com

Read Time Series Forecasting Using Foundation Models on liveBook Get the book at manning.com

Related concepts

• decoder-only transformeruses · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
• Moiraiequivalent to · Moirai is a foundation model designed for time-series forecasting across varied datasets.
• patch lengthconfigures · Patch length is the number of observations contained in each time-series patch.
• mixture of expertsuses · A mixture-of-experts model is a neural-network architecture with specialized expert subnetworks and a routing mechanism that selects or combines experts for each input.
• patchinguses · Patching is the technique of grouping adjacent time-series points into fixed-length segments that are processed as tokens.
• projectionuses · A projection is a mapping or operation that produces a selected or derived representation of an input, often by retaining specified components or mapping it to a subspace.

This explainer was generated by Manning's knowledge graph from the text of Time Series Forecasting Using Foundation Models. The section links point to the parts of the book it draws on.