chapter two

2 Building a foundation model

 

This chapter covers

  • Exploring the architecture of N-BEATS
  • Pretraining our own model for transfer learning
  • Fine-tuning a pretrained model
  • Understanding the challenges of building a foundation model

In chapter 1, we defined foundation models, discovered their advantages and drawbacks, and explored the transformer architecture, which powers the vast majority of foundation models that we will encounter throughout this book. Before we delve into those advanced models, let’s build our own tiny foundation model to understand the concepts surrounding foundation models and appreciate the challenges researchers overcome to build them.

Although technically, many deep learning models can be pretrained and used for transfer learning, we’ll use the N-BEATS model. This lightweight model is fast to train and particularly effective at generalizing to different time series.

In this chapter, we explore basis expansion and the architecture of N-BEATS before training it on a diverse dataset, effectively reproducing the steps for building a foundation model on a much smaller scale. Here, the main goal is to get hands-on experience building a foundation model and to appreciate the difficulty of this task, rather than to build a performant, large time model.

2.1 Exploring the architecture of N-BEATS

2.1.1 Basis expansion

2.2 Architecture of N-BEATS

2.2.1 A block in N-BEATS

2.2.2 A stack in N-BEATS

2.2.3 Assembling N-BEATS

2.3 Pretraining our model

2.4 Pretraining N-BEATS

2.5 Transfer learning with our pretrained model

2.6 Fine-tuning our pretrained model

2.7 Evaluating each approach

2.8 Forecasting at another frequency

2.9 Understanding the challenges of building a foundation model

2.10 Next steps

Summary