1 Understanding foundation models
This chapter covers
- Defining foundation models
- Explore the Transformer architecture
- Advantages and drawbacks of using foundation models
- Overview of foundation models for time series forecasting
Foundation models represent a major shift in paradigm for the field of machine learning. Traditionally, we build data-specific models, meaning that each model is trained on a dataset specific to a particular scenario. Thus, the model specializes in a single use-case. In another situation, another model would have to be trained with data specific to that situation.
Here, it is important to distinguish between the model and the algorithm. The algorithm outlines the steps to be performed by a program in order to achieve a goal. A model is the result of applying an algorithm on a dataset.
For example, consider a linear regression. The algorithm stays constant, in the sense that it will always try to create a straight line that minimizes the sum of the squared distances between all points and the line. However, depending on the dataset that we fit on, the model will be completely different, as shown in figure 1.1
Figure 1.1 Result of performing linear regression on two different datasets. While the algorithm to build the linear model stays the same, the model is definitely very different depending on the dataset used.
