Experimentation treats model development as a curiosity-driven investigation: teams test ideas, inspect results, and learn about the data and domain while moving toward a useful model. section 7.4.1
Run experiments inside a managed framework:
- Write experimental details upfront in a lab book or equivalent record. section 7.4.1
- Record model designs, investigation results, and decisions so work remains reproducible and does not needlessly repeat. section 7.4.1
- Protect evaluation from data leakage by preventing test information from entering training or indirectly driving optimization. section 7.4.1
- Frame projects as experiments or investigations when data availability, data quality, or implementation feasibility remains uncertain. section 3.2.3
Validation-driven overfitting
A team repeatedly optimizes models because one performs well on validation data. If those choices exploit quirks in that data, performance can disappoint in the real world; documenting the investigation and protecting test data helps expose the risk. section 7.4.1
