A token is a discrete representation of part of a time series. Depending on the model, it may represent a unique value, a window or patch of values, or fixed-context past time steps combined with date-time features. section 1.2.1 section 4.1.1
Common token construction strategies include:
- Represent raw observations as individual values or windows of values before learning an embedding. section 1.2.1
- Combine lagged values over a fixed context length with date-time features such as hour-of-day, day-of-week, and quarter-of-year. section 4.1.1
- Convert continuous values into a finite vocabulary through mean scaling and quantization. section 5.2 section 5.3
- Treat a token as a patch containing several time steps, so autoregressive decoding can forecast groups rather than individual observations. section 6.2.2
After tokenization, transformer models commonly project tokens into hidden dimensions and add positional encoding so attention can distinguish order and repeated values. Some models instead train on finite discrete tokens, then dequantize and unscale generated tokens back into predictions. section 1.2.1 section 4.1.1 section 5.2
Hourly forecasting token
For hourly data, a token can contain a fixed history of lagged observations plus calendar features. Hour-of-day changes at every step, while day-of-week and day-of-month change less often, giving the model clues about the series frequency. section 4.1.1
Figure 4.2 Tokenization process in Lag-Llama. Here, the token of the current time step includes lagged values of the series over a certain context length, as well as date-time features.
Visualizes how a current-time-step token combines lagged values over a context window with date-time features. section 4.1.1
