A baseline model is a reference for judging whether an adapted model performs better on its intended task. It may be the model before fine-tuning or a different model already proficient at a similar task. The comparison helps quantify improvement rather than treating a completed training run as proof of success. section 9.3.2
Make the comparison useful:
- Identify where default models fall short, define task-specific evaluation criteria, and establish a baseline before fine-tuning. section 9.2
- After training, compare the adapted model with both the base model and the task baseline using quantitative and qualitative measures. section 9.2
- Choose metrics for the application, supplement automated scores with manual review and user feedback, and log production performance over time. section 9.3.2
Translation quality
For a fine-tuned translation model, compare translation accuracy or fluency with a standard translation model. The baseline shows whether adaptation improved the task that matters, not merely whether training produced a new model. section 9.3.2
Treat a single improved training metric as a signal, not a deployment decision. Mean token accuracy, for example, does not show how well a model generalizes to unseen data. Its meaning depends on the task, so consider other metrics and qualitative evaluation alongside the baseline comparison. section 9.3.4
