A training dataset is a collection of examples used to fit a machine-learning model’s parameters. Its quality matters because the model can learn patterns that reflect the data’s limitations rather than the real-world risk. section 8.5.2
Loan underwriting
Imagine a model that estimates whether a residential-loan applicant will default. If its training dataset comes mainly from an economic downturn, it may learn an exaggerated relationship between age and default, producing results that do not represent long-term risk. section 8.5.2
Treat a training dataset as evidence to inspect, not as an automatically neutral record of reality:
- Check whether the examples reflect the intended use case and whether important groups or situations are underrepresented. section 8.5.2
- Recognize that a supposedly original, unbiased dataset may be difficult to define when few real-world examples exist. section 8.5.2
- Disclose any output adjustments or post-processing, and treat them as no substitute for developing a better model. section 8.5.2
Training data also helps explain why generative AI can produce plausible but inaccurate or outdated answers: the model follows patterns in its content corpus, whose coverage and cutoff constrain its knowledge. section 1.1
