Normalization adjusts feature values so their distributions make sense for the intended outcome. It belongs alongside schema design, anomaly checks, tokenization, and embedding when preparing data for model training. section 4.3.2
Normalization also forms part of the architecture of a model reimplemented from scratch: it is one of the main differences between Llama 3 and a simple feed-forward network, alongside attention, activation layers, and layer count. section 9.5
A practical preparation workflow is:
- Define a schema for the features. section 4.3.2
- Normalize or scale features into distributions that suit the desired outcome. section 4.3.2
- Check for bias and anomalies instead of relying only on automated checks. section 4.3.2
- Convert the prepared data into model-ingestible representations such as tokens and embeddings, then train, check, and retrain. section 4.3.2
