A decoder-only transformer generates a forecast autoregressively: it predicts one token, uses that token to inform the next, and omits an encoder component. In time-series models, each token can represent a patch of time steps rather than one individual value. section 6.2.2
In forecasting architectures, the design has several practical consequences:
- Generates forecast patches sequentially, so longer horizons require repeated autoregressive steps. section 6.2.2
- Uses patching to reduce token counts; larger output patches can reduce the number of steps needed for long-horizon forecasts. section 7.1.1
- Can add specialized components after self-attention, such as a gating function and sparse mixture-of-experts layer. section 6.2 section 6.2.2
- Supports either probabilistic outputs, such as prediction distributions, or deterministic point forecasts, depending on the surrounding model. section 4.1.2 section 7.0
TimesFM and Moirai-MoE
Moirai-MoE generates forecast tokens one patch at a time and routes them through at most two of 32 specialized experts. TimesFM instead uses residual blocks around a decoder-only transformer; with 32-step input patches and 128-step output patches, a 256-step forecast requires only two autoregressive steps. section 6.2.2 section 7.1.1
Figure 7.1 Architecture of TimesFM. The input series is patched and set to a residual block that maps the input patches to a vector. The residual block is a multilayer perceptron with one hidden layer and a skip connection. The output vector is added to positional encoding and sent to the decoder-only transformer. The output of the transformer is sent to another residual block that returns a sequence of predicted values.
Use the architecture diagram to trace patching, residual-block embeddings, positional encoding, decoder-only processing, and output projection. section 7.1.1
