Lag-Llama

As explained in Time Series Forecasting Using Foundation Models

Lag-Llama is a foundation model designed for probabilistic time-series forecasting using lagged series values and date-time features.

Lag-Llama is an open-source foundation model for zero-shot probabilistic time-series forecasting. It predicts a distribution of future values rather than a single point forecast, allowing prediction intervals to be selected from the output. section 4.1 section 4.6

Its decoder-only architecture is based on LLaMA. Tokenization combines lagged series values over a context window with date-time features, while a distribution head produces future values autoregressively. The model assumes a Student’s t-distribution for its predictions. section 4.1.1 section 4.1.2

Figure 4.1 General architecture of Lag-Llama. It features only the decoder portion of the transformer architecture; the distribution head is responsible for generating the distribution of future values.

Use the architecture diagram to follow tokenization, linear projection, decoder attention, distribution generation, and autoregressive feedback. section 4.1.1

Important operating constraints shape when Lag-Llama is a practical choice:

• Handles one univariate series at a time and does not accept exogenous features. section 4.0
• Was pretrained on daily, hourly, 15-minute, and 1-minute data, but not weekly, monthly, or quarterly data. section 4.1.2
• Usually performs best over horizons of 24 to 60 time steps; longer horizons may reduce prediction quality. section 4.1.2
• Requires cloning the repository, installing dependencies, downloading weights, and using GluonTS for inference rather than a packaged Python API. section 4.2 section 4.3.1

Local inference and fine-tuning setup

A practical setup loads lag-llama.ckpt, selects a device, sets prediction_length and context_length, constructs a LagLlamaEstimator, and fine-tunes with parameters such as lr=5e-4, batch_size=64, and max_epochs=50. A context length of 64 is used after producing the best observed performance in the exercise. section 4.3.1

Figure 4.2 Tokenization process in Lag-Llama. Here, the token of the current time step includes lagged values of the series over a certain context length, as well as date-time features.

Use the tokenization figure to inspect how each token combines a context of lagged values with date-time features at multiple frequencies. section 4.1.1

How Time Series Forecasting Using Foundation Models covers it

How deep it goes

Explains architecture, tokenization, pretraining coverage, setup, and fine-tuning while stopping short of production deployment.

Examples

Uses architecture and tokenization diagrams plus repository setup, checkpoint loading, estimator configuration, and fine-tuning code.

What it assumes

Assumes Python environment setup and time-series data formatting, while introducing probabilistic forecasting and the model architecture.

What you can do after

Apply it via a cloned repository and GluonTS to generate probabilistic forecasts or fine-tune a local checkpoint.

How it connects

Connects decoder-only transformers, LLaMA, probabilistic forecasting, Student’s t-distributions, lag features, date-time features, and GluonTS.

liveBookmanning.com

Read Time Series Forecasting Using Foundation Models on liveBook Get the book at manning.com

Related concepts

• PandasDatasetuses · PandasDataset is a GluonTS dataset representation that stores time-series data in a format compatible with GluonTS models.
• decoder-only transformeruses · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
• neuralforecastcontrasts with · NeuralForecast is an object in the neuralforecast library that manages data windows, batching, and model training.
• GluonTSuses · GluonTS is an open-source Python toolkit for building, training, and evaluating probabilistic time-series models.
• tokenizationuses · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
• student's t-distributionuses · Student’s t-distribution is a family of continuous probability distributions with heavier tails than the normal distribution, used in inference when variability is estimated from data.

This explainer was generated by Manning's knowledge graph from the text of Time Series Forecasting Using Foundation Models. The section links point to the parts of the book it draws on.