Experiment tracking
Experiment tracking is the systematic recording and organization of experiment configurations, metrics, artifacts, and results to support inspection, comparison, or reproducibility.
Machine Learning Platform Engineering
An experiment tracker is a software component that records machine-learning experiment configurations, results, and artifacts.
- How it is used
- Organizes exploratory and pipeline training runs so teams can compare results, reproduce configurations, and investigate model behavior.
- How deep it goes
- Works through local MLflow setup, run and artifact logging, then run-linked training and validation; does not detail tracker internals.
- Examples
- Uses income-classification notebook plots and model runs, then recommender runs that compare learning-rate settings and validation metrics.
- What you can do after
- Apply it via MLflow to group and compare runs, preserve artifacts, and load a model by run ID for validation.
Read how Machine Learning Platform Engineering explains experiment tracking →
LLMs in Production
An experiment tracker is a tool that records machine-learning experiments, configurations, artifacts, and results so they can be inspected and compared.
- How it is used
- Supports shared, searchable recording and comparison of machine-learning tests within an MLOps platform.
- How deep it goes
- Explains purpose, collaboration, fine-tuning lineage, evaluation tooling, and registry integration, but does not provide implementation steps.
- Examples
- Uses notebooks, MLflow, CometML, Weights & Biases, checkpoints, training experiments, and evaluation metrics as concrete examples.
- What you can do after
- Diagnose production failures, compare experiments, preserve lineage, and reduce repeated mistakes through searchable records.
Machine Learning System Design
The process of organizing, tracking, preserving, and overseeing experiments and their results.
- How it is used
- Organizes experiments, artifacts, metrics, and reports so teams can compare results, reproduce runs, and assess training outcomes.
- How deep it goes
- Moves from notebooks for solo researchers to centralized repositories, reproducibility controls, artifact preservation, and failure investigation.
- Examples
- Uses notebooks, centralized repositories, validation reports, and an incident involving overwritten model results.
- What you can do after
- Diagnose/debug with preserved experiment records by comparing runs, tracing overwritten results, and checking artifacts before release.
Read how Machine Learning System Design explains experiment tracking →
Transformers in Action
Experiment tracking is the systematic recording and organization of configurations, metrics, artifacts, and results from computational experiments.
- How it is used
- Organizes LLM training and inference development through run logging, resource monitoring, comparison, and shareable reports.
- How deep it goes
- Explains logging, resource analysis, visualization, and sweep configuration, but does not detail tracking-system internals.
- Examples
- Uses W&B reports, GPU-utilization analysis, sweep configurations, dashboards, parallel-coordinate plots, and sortable run tables.
- What you can do after
- Operate W&B to compare runs, monitor resources, share reports, and search hyperparameters across training experiments.
Read how Transformers in Action explains experiment tracking →
Delivering Machine Learning Projects: From design to deployment
Experiment tracking is the systematic recording of machine learning experiment parameters, inputs, outputs, and performance results.
- How it is used
- Supports disciplined model development by making runs comparable, traceable, and reviewable.
- How deep it goes
- Explains recorded metadata, API-based persistence, retrieval, and repository links, but not tracking-system internals.
- Examples
- Uses linear-model runs comparing penalties, test-set sizes, durations, l1_ratio values, and AUC scores.
- What you can do after
- Apply it via MLflow to record, retrieve, compare, and explain model experiments.
Related concepts
- MLflowuses · MLflow is an open-source platform for managing the machine-learning lifecycle, including experiment tracking, model and artifact management, and model deployment.
- Weights and Biasesuses · Weights & Biases is a machine-learning development platform for tracking experiments and managing runs, metrics, artifacts, models, and related workflows.
- mlopsused by · MLOps is the discipline of applying engineering and operational practices to develop, deploy, monitor, maintain, and manage machine-learning systems throughout their life cycle.
- model registryused by · A model registry is a system for storing, organizing, versioning, and managing machine-learning models and their associated metadata throughout their lifecycle.
- performanceuses · Performance is the measurable effectiveness or operational behavior of a system under specified conditions, assessed through relevant qualities such as task results, speed, responsiveness, throughput, or resource use.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.