Model evaluation measures a trained model on a separate holdout dataset. In a production workflow, that dataset should reflect the diversity and characteristics of real inputs without leaking information from training. Results determine whether to proceed or loop back to the model architecture, data quality, or data collection. section 2.1.4 section 1.1.1
Make evaluation a repeatable pipeline stage:
- Choose metrics that match the task and business goal. Classification may require different precision–recall trade-offs; regression may use mean absolute error or mean squared error. section 2.1.4
- Automate runs and centrally track results. Version and trace the holdout data as carefully as training data; a changed split can introduce leakage and require reevaluation and retraining. section 2.1.4
- Inspect errors, not just aggregate scores. Look for patterns in misclassified samples and test robustness with edge cases or adversarial inputs; feed findings back into data collection. section 2.1.4
The metric depends on the cost of an error
A medical classifier may prioritize false-positive rate and precision to minimize harm. A financial application may value recall to avoid missing an event. Neither choice follows from model accuracy alone; the evaluation target must reflect the intended decision. section 2.1.4
LLM applications need a different test criterion. Sampling can produce several valid responses to the same prompt, so an exact-output assertion may fail even when the answer is acceptable. Evaluate semantic correctness, factual accuracy, guideline adherence, and safety instead. DeepEval is identified as a tool for semantic evaluation; adversarial testing also helps probe prompt injection and scope violations before deployment. section 12.1.1 section 13.0 section 13.5
