Any-variate attention mechanism

As explained in Time Series Forecasting Using Foundation Models

An any-variate attention mechanism enables attention over flattened sequences containing an arbitrary number of input variates.

An any-variate attention mechanism enables attention over flattened sequences containing any number of input series and exogenous features. Flattening removes the fixed-dimensionality constraint: adding or removing a feature changes sequence length rather than model dimensionality. section 6.1.1

The mechanism distinguishes series during attention:

• Flatten multivariate inputs into one long sequence for the encoder. section 6.1.1
• Associate each token with a variate index. section 6.1.1
• Add a learned binary attention bias that uses one parameter for tokens from the same series and another for different series. section 6.1.1

Figure 6.3 To handle an arbitrary number of variates, Moirai flattens the data into a single series.

Illustrates how multiple variates become one flattened sequence before attention. section 6.1.1

How Time Series Forecasting Using Foundation Models covers it

How deep it goes

Explains flattening, variate indexing, and learned same- versus cross-series attention bias without deriving the full equations.

Examples

Uses a target series with two exogenous features as the flattening example.

What you can do after

Understand how to adapt attention to forecasting inputs with arbitrary numbers of variates.

How it connects

Connects multivariate time series, exogenous features, attention mechanisms, and encoder-only transformers.

liveBookmanning.com

Read Time Series Forecasting Using Foundation Models on liveBook Get the book at manning.com

Related concepts

• binary attention biasimplemented by · A binary attention bias is a learned additive bias that distinguishes attention between positions from the same variate and positions from different variates.

This explainer was generated by Manning's knowledge graph from the text of Time Series Forecasting Using Foundation Models. The section links point to the parts of the book it draws on.