liveBook · Manning

Reproducibility

As explained in AI Model Evaluation

Reproducibility is the ability to repeat an evaluation and obtain comparable results from the same documented inputs, code, configuration, and procedures.

Reproducibility means being able to repeat an evaluation and obtain comparable results from documented inputs, code, configuration, and procedures. Versioning makes it possible to revisit results and track how changes to a model, data, prompts, or evaluation setup affect them. Without it, inconsistent runs can erode trust. section 2.2.2

Version and document the parts needed to reconstruct an evaluation:

  • Datasets and sampling, model artifacts, feature pipelines, evaluation code, metric definitions, and configuration. section 2.4 section 9.10.2
  • For LLM evaluations, prompt templates, system instructions, retrieved context, judge models and prompts, and grading rubrics. section 2.4 section 2.2.2
  • For agent evaluations, tool schemas and availability, workflow definitions, and traces of tool calls or intermediate steps. section 2.4

A reusable setup also needs logging, reliable data pipelines, and traceable outputs. Dashboards or reports make comparisons interpretable; logging should capture enough detail without creating bottlenecks or privacy risks. DVC can track datasets, model artifacts, and pipeline stages, while MLflow supports experiment tracking and model versioning. section 2.4 section 2.2.2

Revisiting an LLM judge evaluation

Record the judge model and configuration, rubric and allowed scores, sampling strategy, and dataset snapshot. The snapshot should identify the exact examples, time window, filters, and model outputs. Keep calibration or human-alignment results too, so a later reviewer can trace and assess the basis for a decision. section 9.10.2

A repeatable result is not automatically a trustworthy measure of quality. Check for evaluation data leakage and, when using an LLM judge, monitor prompt and model changes that can alter judgments. Clear task framing and judgment formats also help make runs consistent and interpretable. section 2.5 section 9.10 section 9.5

How AI Model Evaluation covers it

How deep it goes
Provides artifact and documentation checklists and engineering rationale, without specifying a complete implementation.
Examples
Uses LLM judge run records, including dataset snapshots, sampling strategies, scoring rubrics, and human-alignment checks.
What you can do after
Operate/configure an evaluation workflow with versioned artifacts, documented runs, and traceable outputs.
How it connects
Connects versioning and experiment tracking with offline evaluation, data leakage, and LLM-as-a-judge calibration.

Also explained in

Compare how 11 books explain reproducibility →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.