Reproducibility
Reproducibility is the ability to obtain consistent or comparable results by repeating a process with the same documented inputs, methods, and relevant conditions.
Designing Deep Learning Systems: A guide for software engineers
The ability to repeat model training with the same inputs and obtain comparable results.
- How it is used
- Supports repeatable production training for debugging, troubleshooting, and trustworthy prediction-based decisions.
- How deep it goes
- Explains reproducibility through versioned datasets, tracked run inputs, workflow state, and production service responsibilities.
- Examples
- Uses versioned ImageNet experiments, static dataset snapshots, and an annotated Metaflow data-processing workflow.
- What you can do after
- Operate/configure reproducible training by versioning data and recording code, hyperparameters, workflow state, models, and artifacts.
Read how Designing Deep Learning Systems: A guide for software engineers explains reproducibility →
Developer Experience on Kubernetes
Reproducibility is the ability to obtain equivalent software or environment results from the same declared inputs.
- How it is used
- Supports consistent development environments and repeatable container-image builds for team workflows, security, and compliance.
- How deep it goes
- Explains input drift in Devcontainers and Dockerfiles, then demonstrates timestamp control as a Buildpacks reproducibility mechanism.
- Examples
- Uses apt-installed httpie, a devcontainer.json configuration, and an Appointments image built with Pack.
- What you can do after
- Evaluate/understand when declared environment and image inputs can produce changing outputs, and apply version pinning or Buildpacks accordingly.
Read how Developer Experience on Kubernetes explains reproducibility →
Infrastructure as Code, Patterns and Practices: With examples in Python and Terraform
Repeatability is the property of automation that produces consistent results when run multiple times.
- How it is used
- Uses configuration as the source of truth for creating, sharing, changing, and scaling infrastructure environments.
- How deep it goes
- Explains configuration drift, version control, immutability, parameterization, modules, and on-demand environments, without detailing a specific IaC implementation.
- Examples
- Uses development-environment recreation, network address changes, shared modules, versioned commits, and on-demand QA environments.
- What you can do after
- Evaluate IaC for reproducibility and apply version-controlled, parameterized configurations to recreate and remove environments.
Data-Oriented Programming: Reduce software complexity
Reproducibility is the ability to recreate the behavior of a computation in another environment using the same relevant inputs and conditions.
- How it is used
- Supports debugging and unit testing by replaying captured function contexts outside the original endpoint or environment.
- How deep it goes
- Explains deterministic arguments, serialization, REPL replay, test fixtures, and external-data limits, but does not develop formal theory.
- Examples
- Uses a nested book catalog, captured JSON arguments, REPL experiments, and unit tests for matching queries.
- What you can do after
- Diagnose/debug with it by capturing inputs, replaying calls, refining behavior, and converting scenarios into repeatable tests.
Read how Data-Oriented Programming: Reduce software complexity explains reproducibility →
Delivering Machine Learning Projects: From design to deployment
Reproducibility is the ability to recreate a computational result from the same specified inputs, components, and procedures.
- How it is used
- Supports rebuilding ML models and reproducing tests across experimental and delivery workflows.
- How deep it goes
- Defines reproducibility through artifact tracking and documents experimental and testing practices, without specifying a particular implementation.
- Examples
- Uses component replacement and a lab-book analogy to illustrate model recreation and experimental records.
- What you can do after
- Evaluate whether a model’s recorded components and testing procedures support recreation and investigation.
Read how Delivering Machine Learning Projects: From design to deployment explains reproducibility →
AI Model Evaluation
Reproducibility is the ability to repeat an evaluation and obtain comparable results from the same documented inputs, code, configuration, and procedures.
- How it is used
- Supports repeatable offline evaluations and traceable comparisons as models, data, and evaluation setups change.
- How deep it goes
- Provides artifact and documentation checklists and engineering rationale, without specifying a complete implementation.
- Examples
- Uses LLM judge run records, including dataset snapshots, sampling strategies, scoring rubrics, and human-alignment checks.
- What you can do after
- Operate/configure an evaluation workflow with versioned artifacts, documented runs, and traceable outputs.
Machine Learning Platform Engineering
Reproducibility is the ability to obtain consistent results when repeating an operation under the same inputs, configuration, and environment.
- How it is used
- Supports reliable ML experimentation, deployment, auditing, and iteration by keeping data, code, configurations, and artifacts aligned.
- How deep it goes
- Explains versioning, automation, modular pipelines, and environment consistency, while deferring detailed algorithmic training mechanics.
- Examples
- Uses labeled dataset splits, automated experiments, Kubeflow preparation pipelines, YOLO training, and versioned RAG configurations.
- What you can do after
- Build it from scratch as a tracked, modular training workflow that executes consistently across environments.
Read how Machine Learning Platform Engineering explains reproducibility →
Machine Learning System Design
Reproducibility is the property of being able to recreate a dataset or computational result from its original source and documented procedure.
- How it is used
- Regenerates datasets reliably during data gathering, preprocessing, updates, and later project work.
- How deep it goes
- Defines documented and tested regeneration procedures, but stops short of prescribing a particular implementation.
- Examples
- Uses manually crafted datasets, omitted preprocessing steps, and regenerated batches as practical examples.
- What you can do after
- Build it from scratch through a documented, tested procedure using the original data source.
Read how Machine Learning System Design explains reproducibility →
Kubernetes for Developers
Reproducibility is the ability to recreate the same application environment from a recorded set of build instructions and inputs.
- How it is used
- Uses containers and declarative Kubernetes configuration to recreate development, staging, and production environments.
- How deep it goes
- Explains recorded build steps and contrasts declarative configuration with imperative commands, without detailing full GitOps practice.
- Examples
- Uses a pseudocode Dockerfile and timeserver deployment and exposure commands as concrete reproducibility examples.
- What you can do after
- Apply declarative Kubernetes configuration across environments and record container construction steps for repeatable deployment.
Read how Kubernetes for Developers explains reproducibility →
Architecting an Apache Iceberg Lakehouse
Reproducibility is the ability to obtain consistent analytical or computational results by repeating a process with the same inputs, versions, and conditions.
- How it is used
- Supports repeatable model training, policy reporting, historical analysis, and disaster recovery across lakehouse environments.
- How deep it goes
- Explains snapshot reads, metadata capture, replication, and path rewriting, but stops short of specifying a complete recovery runbook.
- Examples
- Uses notebook-based training workflows, MLflow outputs, cross-region backups, and preserved public-sector reporting views.
- What you can do after
- Apply snapshot-based reads and metadata controls to repeat training, reproduce historical reports, and reconstruct replicated tables.
Read how Architecting an Apache Iceberg Lakehouse explains reproducibility →
Build AI Drug Discovery Pipelines
Reproducibility is the ability to obtain consistent results by repeating an analysis with the same documented inputs, software, models, and procedures.
- How it is used
- Supports reliable repetition and validation of multimodal, end-to-end drug-discovery analyses.
- How deep it goes
- Specifies tracking models, datasets, preprocessing, and inference parameters, while emphasizing validation rather than detailing an implementation checklist.
- Examples
- Compares analyses using ESM-2 versions 1.0 and 2.0 and considers cascaded structure-prediction and docking accuracy.
- What you can do after
- Operate reproducible analyses by recording component versions, datasets, preprocessing, inference parameters, and end-to-end validation results.
Read how Build AI Drug Discovery Pipelines explains reproducibility →
Also covered in
- Data Without Labels
- Data Pipelines with Apache Airflow, Second Edition
- Effective Data Analysis
- The Well-Grounded Data Analyst
- Terraform in Depth: Infrastructure as Code with Terraform and OpenTofu
- AI Agents in Action, Second Edition
- Data-Oriented Programming in JavaIn early access (MEAP)
- Machines that ThinkIn early access (MEAP)
- Google Anthos in Action
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- automationAutomation is the execution of tasks or processes by machines or software with reduced direct human intervention.
- area under the ROC curveArea under the ROC curve is a classification metric that measures how well a scoring model ranks positive examples above negative examples across decision thresholds.
- complianceCompliance is conformity with applicable legal, regulatory, contractual, policy, or other governing requirements.
Related concepts
- versioningproduced by · Versioning is the practice of assigning or maintaining identifiable revisions of artifacts over time so their changes, histories, or compatibility can be tracked.
- checklistused by · A structured list of tasks, questions, or criteria used to guide an activity or systematically verify that relevant requirements have been addressed.
- infrastructure as codepart of · Infrastructure as code is the practice of defining, provisioning, and managing computing infrastructure through machine-readable code or configuration rather than manual procedures.
- random seedconfigured by · A random seed is a value that initializes a pseudorandom process and determines its subsequent sequence of outputs.
- snapshotaddressed by · A snapshot is a fixed representation of data or system state as it existed at a particular point in time.
Questions about reproducibility
What is reproducibility?
Reproducibility is the ability to obtain consistent or comparable results by repeating a process with the same documented inputs, methods, and relevant conditions.
Which Manning books explain reproducibility?
Designing Deep Learning Systems: A guide for software engineers, Developer Experience on Kubernetes, Infrastructure as Code, Patterns and Practices: With examples in Python and Terraform, Data-Oriented Programming: Reduce software complexity, Delivering Machine Learning Projects: From design to deployment, AI Model Evaluation, Machine Learning Platform Engineering, Machine Learning System Design and 3 more.
What should I know before learning reproducibility?
Manning books that teach reproducibility usually introduce language model, automation, area under the ROC curve and compliance first.
What is reproducibility part of?
Reproducibility is part of infrastructure as code. Infrastructure as code is the practice of defining, provisioning, and managing computing infrastructure through machine-readable code or configuration rather than manual procedures.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.










