2 Building a foundation model

 

This chapter covers

  • Exploring the architecture of N-BEATS
  • Pretraining our own model for transfer learning
  • Fine-tuning a pretrained model
  • Understanding the challenges of building a foundation model

In the previous chapter, we defined foundation models, discovered their advantages and drawbacks, and explored the Transformer architecture in detail, which powers the vast majority of foundation models that we will encounter throughout this book. Before we dive into those advanced models, let’s build our very own tiny foundation model to better understand the concepts surrounding foundation models and appreciate the challenges researchers overcome to build them.

While many deep learning models can technically be pretrained and used for transfer learning, we specifically use the N-BEATS model. This is a lightweight model that is fast to train, and it was shown to be particularly effective at generalizing to different time series.

In the following sections, we discover the concept of basis expansion and the architecture of N-BEATS, before training it on a diverse dataset, effectively reproducing the steps for building a foundation model but on a much smaller scale. Here, the goal is mostly to get hands-on experience with the concepts behind building a foundation model and appreciate the difficulty of this task, rather than build a performant large time model.

2.1 Explore the architecture of N-BEATS

2.1.1 Basis expansion

2.1.2 Architecture of N-BEATS

2.2 Pretraining our model

2.2.1 Pretraining N-BEATS

2.3 Transfer learning with our pretrained model

2.4 Fine-tuning our pretrained model

2.5 Evaluating each approach

2.6 Forecasting at another frequency

2.7 Understand the challenges of building a foundation model

2.8 Next steps

2.9 Summary