Categorical data takes values from a predefined set of discrete categories rather than a continuous numeric scale. A token, for example, is a categorical feature: represent it with dimensions in a feature space or as a category vector. This representation handles the token itself; representing token order requires a separate modeling choice. section 14.3
Choose the representation according to whether relationships between values or their order matters:
- Use feature-space dimensions or category vectors for individual categorical values. section 14.3
- Use a bag-of-words model when treating text as an unordered set of categorical tokens. section 14.3
- Use a sequence model when token order matters; recurrent models process tokens in order, while Transformers add position information. section 14.3
- Use densely connected networks for vector inputs containing numerical or categorical attributes when no specific input structure is assumed. section 20.1.6
Categorical outputs in a dense network
For single-label categorical classification, finish a Dense network with one unit per class and softmax activation. Pair one-hot targets with categorical_crossentropy, or integer targets with sparse_categorical_crossentropy. For multilabel classification, use one sigmoid unit per class, k-hot targets, and binary_crossentropy. section 20.1.6
Table 20.1 Model architectures for different data types
| Input |
Output |
Model |
|---|---|---|
| Vector data |
Class probability, Regression value |
Densely connected network |
| Timeseries data |
Class probability, Regression value |
RNN, Transformer |
| Images |
Class probability, Regression value |
ConvNet |
| Text |
Class probability, Regression value |
Transformer |
| Text, Images |
Text |
Transformer |
| Text, Images |
Images |
VAE, Diffusion Model |
Compare model architectures by input type and prediction output: vector data maps to densely connected networks, while timeseries, images, and text call for specialized architectures. section 20.1.6
