Time-series data are sequential observations whose order carries meaning. A transformer converts the raw series into tokens, learns multidimensional embeddings, and adds positional encoding so repeated values remain distinguishable and future information does not leak into the past. section 1.2.1
Important implementation characteristics include:
- Tokens may represent individual values or windows of values. section 1.2.1
- Fixed sinusoidal positional encoding uses sine and cosine at different frequencies, matching the embedding dimensions. section 1.2.1
- Domain diversity exposes forecasting models to different trends and seasonalities, while minimally processed data preserves noise and outliers. section 3.2.1
Temporal patterns
Hourly traffic can show daily and weekly seasonality, while daily weather can show yearly seasonality. These patterns illustrate why ordering and domain coverage matter. section 3.2.1
Figure 1.3 Visualizing the result of feeding a time series through an embedding layer. First, the input is tokenized, and an embedding is learned. The result is the abstract representation of the input made by the model.
Visualizes tokenization, learned embeddings, and the abstract representation passed onward. section 1.2.1
