Cover of Time Series Forecasting Using Foundation Models

What you will learn in Time Series Forecasting Using Foundation Models

How to build high accuracy predictive models

by Marco Peixeiro

Read it on liveBook Get the book at manning.com

Chapter by chapter

Chapter 1: Understanding foundation models

1.1 Defining a foundation model · 1.2 Exploring the transformer architecture · 1.2.1 Feeding the encoder · 1.2.2 Inside the encoder · 1.2.3 Making predictions · 1.3 Advantages and disadvantages of foundation models · 1.3.1 Benefits of foundation forecasting models · 1.3.2 Drawbacks of foundation forecasting models · 1.4 Next steps · 1.5 Summary

What you learn in chapter 1 →

Chapter 2: Building a foundation model

2.1 Exploring the architecture of N-BEATS · 2.1.1 Basis expansion · 2.2 Architecture of N-BEATS · 2.2.1 A block in N-BEATS · 2.2.2 A stack in N-BEATS · 2.2.3 Assembling N-BEATS · 2.3 Pretraining our model · 2.4 Pretraining N-BEATS · 2.5 Transfer learning with our pretrained model · 2.6 Fine-tuning our pretrained model · 2.7 Evaluating each approach · 2.8 Forecasting at another frequency · 2.9 Understanding the challenges of building a foundation model · 2.10 Next steps · 2.11 Summary

What you learn in chapter 2 →

Chapter 3: Forecasting with TimeGPT

3.1 Defining generative pretrained transformers · 3.2 Exploring TimeGPT · 3.2.1 Training TimeGPT · 3.2.2 Quantifying uncertainty in TimeGPT · 3.3 Forecasting with TimeGPT · 3.3.1 Initial setup · 3.3.2 Zero-shot forecasting · 3.3.3 Performance evaluation · 3.4 Fine-tuning with TimeGPT · 3.4.1 Fine-tuning TimeGPT · 3.4.2 Evaluating the fine-tuned model · 3.4.3 Controlling the depth of fine-tuning · 3.5 Forecasting with exogenous variables · 3.5.1 Preparing the exogenous features · 3.5.2 Forecasting with exogenous variables · 3.5.3 Explaining the effect of exogenous features with Shapley values · 3.5.4 Evaluating forecasts with exogenous features · 3.6 Cross-validating with TimeGPT · 3.7 Forecasting on a long horizon with TimeGPT · 3.8 Detecting anomalies with TimeGPT · 3.8.1 Detecting anomalies · 3.8.2 Evaluating anomaly detection · 3.9 Next steps · 3.10 Summary

What you learn in chapter 3 →

Chapter 4: Zero-shot probabilistic forecasting with Lag-Llama

4.1 Exploring Lag-Llama · 4.1.1 Viewing the architecture of Lag-Llama · 4.1.2 Pretraining Lag-Llama · 4.2 Forecasting with Lag-Llama · 4.2.1 Setting up Lag-Llama · 4.2.2 Zero-shot forecasting with Lag-Llama · 4.2.3 Changing the context length in Lag-Llama · 4.3 Fine-tuning Lag-Llama · 4.3.1 Handling initial setup · 4.3.2 Reading and splitting the data in Colab · 4.3.3 Launching the fine-tuning procedure · 4.3.4 Forecasting with a fine-tuned model · 4.3.5 Evaluating the fine-tuned model · 4.4 Model comparison table · 4.5 Next steps · 4.6 Summary

What you learn in chapter 4 →

Chapter 5: Learning the language of time with Chronos

5.1 Discovering the T5 family · 5.2 Exploring Chronos · 5.3 Using tokenization in Chronos · 5.4 Training a model with Chronos · 5.4.1 Tackling data scarcity with augmentation techniques · 5.4.2 Examining the pretrained Chronos models · 5.4.3 Selecting the appropriate Chronos model · 5.5 Forecasting with Chronos · 5.5.1 Initial setup · 5.5.2 Predictions · 5.6 Cross-validating with Chronos · 5.6.1 Running cross-validation · 5.6.2 Evaluating Chronos · 5.7 Fine-tuning Chronos · 5.7.1 Performing initial setup · 5.7.2 Configuring the fine-tuning parameters · 5.7.3 Launching the fine-tuning procedure · 5.7.4 Forecasting with a fine-tuned model · 5.7.5 Evaluating the fine-tuned model · 5.8 Detecting anomalies with Chronos · 5.9 Next steps · 5.10 Summary

What you learn in chapter 5 →

Chapter 6: Moirai: A universal forecasting transformer

6.1 Exploring Moirai · 6.1.1 Viewing the architecture of Moirai · 6.1.2 Pretraining Moirai · 6.1.3 Selecting the appropriate model · 6.2 Discovering Moirai-MoE · 6.2.1 Patching and embedding · 6.2.2 Studying the decoder-only transformer · 6.2.3 Pretraining Moirai-MoE · 6.3 Forecasting with Moirai · 6.3.1 Zero-shot forecasting with Moirai · 6.3.2 Cross-validation with Moirai · 6.3.3 Forecasting with exogenous features · 6.4 Detecting anomalies with Moirai · 6.5 Next steps · 6.6 Summary

What you learn in chapter 6 →

Chapter 7: Deterministic forecasting with TimesFM

7.1 Examining TimesFM · 7.1.1 Architecture of TimesFM · 7.1.2 Pretraining TimesFM · 7.2 Forecasting with TimesFM · 7.2.1 Zero-shot forecasting with TimesFM · 7.2.2 Cross-validation with TimesFM · 7.2.3 Forecasting with exogenous features · 7.3 Fine-tuning TimesFM and anomaly detection · 7.4 Next steps · 7.5 Summary

What you learn in chapter 7 →

Chapter 8: Forecasting as a language task

8.1 Overview of LLMs and prompting techniques · 8.1.1 Exploring Flan-T5 and Llama-3.2 · 8.1.2 Understanding the basics of prompting · 8.2 Forecasting with Flan-T5 · 8.2.1 Function to forecast with Flan-T5 · 8.2.2 Forecast with Flan-T5 · 8.3 Cross-validation with Flan-T5 · 8.3.1 Running cross-validation · 8.3.2 Evaluating Flan-T5 · 8.4 Forecasting with exogenous features with Flan-T5 · 8.4.1 Including exogenous features with Flan-T5 · 8.4.2 Extracting future values of exogenous variables · 8.4.3 Cross-validating with external features · 8.4.4 Evaluating Flan-T5 forecasts with exogenous features · 8.5 Detecting anomalies with Flan-T5 · 8.5.1 Defining a function for anomaly detection with Flan-T5 · 8.5.2 Running anomaly detection · 8.6 Forecasting with Llama-3.2 · 8.6.1 Performing initial setup · 8.6.2 Creating a function to forecast via API call · 8.6.3 Making predictions · 8.7 Cross-validating with Llama-3.2 · 8.8 Detecting anomalies with Llama-3.2 · 8.8.1 Modifying the system prompt · 8.8.2 Defining a function for anomaly detection · 8.8.3 Running anomaly detection with Llama-3.2 · 8.8.4 Evaluating anomaly detection · 8.9 Next steps · 8.10 Summary

What you learn in chapter 8 →

Chapter 9: Reprogramming an LLM for forecasting

9.1 Discovering Time-LLM · 9.1.1 Patch reprogramming · 9.1.2 Discovering Prompt-as-Prefix · 9.1.3 Making predictions · 9.2 Forecasting with Time-LLM · 9.2.1 Performing initial setup · 9.2.2 Generating forecasts · 9.3 Cross-validating with Time-LLM · 9.4 Evaluating Time-LLM · 9.5 Detecting anomalies with Time-LLM · 9.5.1 Detecting anomalies · 9.5.2 Evaluating anomaly detection · 9.6 Next steps · 9.7 Summary

What you learn in chapter 9 →

Chapter 10: Capstone project: Forecasting daily visits to a blog

10.1 Introducing the use case · 10.2 Walking through the project · 10.2.1 Setting the constants · 10.2.2 Forecasting with a seasonal naïve model · 10.2.3 Forecasting with ARIMA · 10.2.4 Forecasting with TimeGPT · 10.2.5 Forecasting with Chronos · 10.2.6 Forecasting with Moirai · 10.2.7 Forecasting with TimesFM · 10.2.8 Forecasting with Time-LLM · 10.2.9 Evaluating all models · 10.3 Staying up to date

What you learn in chapter 10 →

Concepts explained in this book

foundational modelA foundation model is a machine learning model trained on large, diverse data and applicable to a wide variety of tasks, often through adaptation or fine-tuning.
fine-tuningFine-tuning is additional training of a pretrained model on task- or domain-specific data by updating some or all of its parameters.
MoiraiMoirai is a foundation model designed for time-series forecasting across varied datasets.
language modelA large language model is a neural-network model trained on large text corpora to perform language understanding and generation tasks.
TimeGPTTimeGPT is a proprietary foundation model designed for time-series forecasting and related time-series analysis.
prediction intervalA prediction interval is a range computed from a forecasting model that is expected to contain a future or unobserved value at a specified confidence level.
TimesFMTimesFM is a foundation model designed for deterministic time-series forecasting.
anomaly detectionAnomaly detection is the task of identifying observations or patterns that substantially deviate from expected behavior.
context windowContext length is the fixed number of preceding sequence elements that a model can consider when processing an input.
zero-shot forecastingZero-shot forecasting is forecasting with a model on a target time series without task-specific training or fine-tuning on that series.
patchingPatching is the technique of grouping adjacent time-series points into fixed-length segments that are processed as tokens.
mean absolute errorMean absolute error is the average of the absolute differences between predicted and actual values.
symmetric mean absolute percentage errorSymmetric mean absolute percentage error is a percentage-based forecasting metric that measures absolute error symmetrically relative to predicted and actual values.
N-BEATSN-BEATS is a neural forecasting model organized as stacks of blocks for modeling time series without requiring explicit series decomposition or input transformations.
transformer architectureA neural-network architecture built from stacked attention and feed-forward processing components, typically organized into encoder and decoder parts.
tokenA token is a discrete representation of a sequence segment, including a fixed-length set of past time steps and associated features.
Lag-LlamaLag-Llama is a foundation model designed for probabilistic time-series forecasting using lagged series values and date-time features.
pretrainingPretraining is initial training of a model on a large dataset to learn general patterns and relationships before applying it to specific tasks.
point forecastDeterministic forecasting is the technique of predicting one value for each future time step without representing forecast uncertainty.
tokenizationTokenization is the process of converting input data into discrete units suitable for sequence-model processing.
decoder-only transformerA decoder-only transformer is a transformer architecture that generates sequence outputs autoregressively from previously generated tokens without an encoder component.
ChronosThe Chronos framework adapts language models into pretrained probabilistic forecasting models by transforming time-series data into a discrete representation.
patch lengthInput patch length is the number of time steps grouped into each input patch.
FLAN-T5Flan-T5 is an instruction-tuned text-to-text transformer model designed to follow natural-language instructions across tasks.
Time-LLMTime-LLM is a language-model-based framework that adapts existing large language models for time-series forecasting.
time-series foundation modelA foundation forecasting model is a broadly pretrained model specialized for forecasting tasks, particularly across time-series data.
forecastingTime-series forecasting is the task of predicting future values of a variable or collection of variables indexed over time.
seasonality effectSeasonality is a recurring pattern in a time series associated with a fixed cycle, such as weekly, daily, or twice-daily repetition.
encoder-only transformerAn encoder-only transformer is a transformer architecture composed of encoder-style sequence-processing layers without a separate decoder.
SARIMASARIMA is a seasonal autoregressive integrated moving-average model for forecasting time series with recurring seasonal structure.
probabilistic modelA probabilistic model represents uncertainty by assigning probability distributions to possible data or prediction outcomes.
data-specific modelA forecasting model designed and trained for a particular dataset, domain, or use case rather than broad general-purpose use.
few-shot learningA few-shot prompt is a model input that includes a small number of examples to demonstrate the desired task or output behavior.
transfer learningTransfer learning is a technique that applies knowledge learned by a model on one dataset or task to a different dataset or task.
binary attention biasA binary attention bias is a learned additive bias that distinguishes attention between positions from the same variate and positions from different variates.
mixture modelA mixture distribution is a probability distribution formed by weighting and combining multiple component distributions.
any-variate attention mechanismAn any-variate attention mechanism enables attention over flattened sequences containing an arbitrary number of input variates.
multi-head attentionAn attention mechanism with multiple parallel attention heads that combines information from different representation subspaces.
promptA system prompt is an instruction supplied to a language model that establishes its context, behavior, tone, or response constraints.
patchA patch is a segment of consecutive time-series observations grouped as one model input unit.
conformal predictionConformal prediction is a model-agnostic, distribution-free method for constructing prediction intervals with a specified coverage level.
PandasDatasetPandasDataset is a GluonTS dataset representation that stores time-series data in a format compatible with GluonTS models.
encoderAn encoder is the sequence-processing component of a transformer that receives token representations and transforms them into contextual representations.
f-scoreF1 score is the harmonic mean of precision and recall.
mean scalingMean scaling is a data transformation that divides each value by the mean of the absolute values of all values in a series.
validation windowA validation window is a contiguous held-out time period on which a forecast is generated during cross-validation.
nixtlanixtla_client is a client library for interacting programmatically with Nixtla forecasting services and retrieving forecast-related outputs.
embedding layerAn embedding layer is a neural-network layer that maps input items or tokens to learned numerical vector representations.
chain-of-thought promptingChain-of-thought prompting is a prompting technique that elicits intermediate reasoning steps before a model produces an answer.
residual errorA forecast residual is the difference between an actual value and its predicted value.
quantizationQuantization is the conversion of continuous numerical values into a finite set of discrete levels or tokens.
GluonTSGluonTS is an open-source Python toolkit for building, training, and evaluating probabilistic time-series models.
inference performanceInference time is the duration required for a model to generate predictions from its input.
reprogrammingReprogramming is the technique of adapting a pretrained language model's representations or processing so it can perform a different task such as forecasting.
M3 datasetThe M3 dataset is a collection of 3,003 time series from the M3 forecasting competition sampled at yearly, quarterly, and monthly frequencies.
out-of-sample forecastAn out-of-sample forecast is a prediction for future observations that were not included in the model input or fitting data.
predictive distributionA predictive distribution is a probability distribution over possible future values conditional on observed input data.
PyTorch LightningPyTorch Lightning is an open-source deep-learning framework that organizes PyTorch training and deployment code.
conformity scoreA conformity score is a numerical measure of how atypical an observation is relative to a prediction, here computed as the absolute forecast residual.
exchangeabilityExchangeability is a probabilistic property under which reordering a collection of observations does not change their joint likelihood.
student's t-distributionStudent's t-distribution is a probability distribution with heavier tails than a normal distribution, commonly used when estimating uncertain quantities from limited data.
mean squared errorMean squared error is the average of the squared differences between predicted and actual values.
weight decayL2 regularization is a penalty method that discourages large model weights by adding their squared magnitudes to the optimization objective.
Moirai-MoEMoirai-MoE is a Moirai-based universal time-series forecasting model that uses fixed-length, nonoverlapping patches across frequencies.
seasonal naïve modelA seasonal naïve model forecasts each value using an observation from the corresponding position in a previous seasonal cycle.
pretrained modelA pretrained model is a model whose parameters were learned on earlier data before its application to a new task or dataset.
neuralforecastNeuralForecast is an object in the neuralforecast library that manages data windows, batching, and model training.
sequenceTime-series data are observations indexed by time, often representing measurements collected sequentially.
data scarcityData scarcity is a condition in which too little suitable data is available for training or analysis.
precisionPrecision is the proportion of observations labeled positive by a detection system that are actually positive.
recallRecall is the proportion of actual positive observations that a detection system correctly identifies.
zero-shot promptingZero-shot prompting is prompting a model to perform a task without providing task-specific examples in the prompt.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.