Categorical encoding represents each discrete category with a vector containing one 1 at the category index and 0s elsewhere. It turns labels or token IDs into numeric features that neural-network layers can ingest. section 4.2.2 section 14.5.1
Encoding classification labels
For labels numbered from 0 through 45, create an array shaped (number_of_labels, 46), then set results[i, label] = 1.0 for each example. NumPy can implement this directly, while Keras provides to_categorical(train_labels). section 4.2.2
Use one-hot encoding with these practical distinctions:
- Encode token ID sequences by giving every token a vocabulary-sized vector with one active position. section 14.5.1
- Expect sparse, binary, high-dimensional features: a 30,000-word vocabulary produces vectors with 30,000 positions. section 14.5.2
- Treat the representation as an independence assumption: distinct categories receive orthogonal vectors with no learned similarity. section 14.5.2
- Match the label representation to the loss: categorical_crossentropy expects categorical labels, whereas integer labels use sparse_categorical_crossentropy. section 4.2.6
