A training dataset is the collection of examples from which a machine-learning model learns parameters or behavior. For LLMs, dataset quality often matters more than parameter count, model architecture, or training-loop cleverness. section 4.3 section 5.5
Treat dataset preparation as an engineering task:
- Acquire, clean, and curate examples before training; remove duplicates and address poor captions or unsuitable content. section 5.5 section 7.2.3
- Inspect how examples are labeled and phrased, because those patterns shape what the model associates and how prompts retrieve it. section 7.2.3 section 7.5
- Balance quality, quantity, and resources: fine-tuning needs less data than training from scratch, while training remains memory intensive. section 5.5
- Review provenance and sensitive content, since model outputs can reproduce outdated, incorrect, copyrighted, licensed, or confidential material present in the data. section 1.3
When labels change the prompt
An image dataset labeled examples with “Asian beauty” but none with “Asian woman.” Knowing that vocabulary enabled more effective prompts and exposed an association embedded in the training data. section 7.2.3 section 7.5
Figure 7.4 Example of DreamBooth from Ruiz et al.7 DreamBooth allows you to finetune an image model to replicate an object’s likeness based on only a few sample input images. Here, with only four example images of a puppy, Dreambooth can put that same dog in many new scenarios.
DreamBooth illustrates how a few carefully chosen sample images can support fine-tuning an image model to reproduce an object in new scenarios. section 7.2.3
