Reproducibility means being able to repeat an evaluation and obtain comparable results from documented inputs, code, configuration, and procedures. Versioning makes it possible to revisit results and track how changes to a model, data, prompts, or evaluation setup affect them. Without it, inconsistent runs can erode trust. section 2.2.2
Version and document the parts needed to reconstruct an evaluation:
- Datasets and sampling, model artifacts, feature pipelines, evaluation code, metric definitions, and configuration. section 2.4 section 9.10.2
- For LLM evaluations, prompt templates, system instructions, retrieved context, judge models and prompts, and grading rubrics. section 2.4 section 2.2.2
- For agent evaluations, tool schemas and availability, workflow definitions, and traces of tool calls or intermediate steps. section 2.4
A reusable setup also needs logging, reliable data pipelines, and traceable outputs. Dashboards or reports make comparisons interpretable; logging should capture enough detail without creating bottlenecks or privacy risks. DVC can track datasets, model artifacts, and pipeline stages, while MLflow supports experiment tracking and model versioning. section 2.4 section 2.2.2
Revisiting an LLM judge evaluation
Record the judge model and configuration, rubric and allowed scores, sampling strategy, and dataset snapshot. The snapshot should identify the exact examples, time window, filters, and model outputs. Keep calibration or human-alignment results too, so a later reviewer can trace and assess the basis for a decision. section 9.10.2
A repeatable result is not automatically a trustworthy measure of quality. Check for evaluation data leakage and, when using an LLM judge, monitor prompt and model changes that can alter judgments. Clear task framing and judgment formats also help make runs consistent and interpretable. section 2.5 section 9.10 section 9.5










