Experiment tracking

As explained in LLMs in Production

An experiment tracker is a tool that records machine-learning experiments, configurations, artifacts, and results so they can be inspected and compared.

Experiment tracking turns trial-and-error into inspectable science by recording tests and results in a shared, searchable system rather than leaving findings in private notebooks. section 3.4.2

Its practical value appears when a production model fails: teams can investigate the original experiment, compare alternatives, and avoid repeating the same mistakes. Trackers sit alongside model registries, feature stores, pipelines, and monitoring in an MLOps platform. section 3.4 section 3.4.2

A useful tracker should support:

• Searchable records of tests and results that others can inspect, not merely manually maintained notebook notes. section 3.4.2
• Links from fine-tuning runs to the starting model checkpoint and, where possible, the original training experiment and data. section 3.4.2
• Evaluation-metric tooling suited to language models, where several metrics may matter simultaneously. section 3.4.2
• Integration with a model registry, since model metadata usually belongs with experiment records and later model versions need operational traceability. section 3.4.3

Choosing a tracker

MLflow is a prominent open-source option with a hosting solution; CometML and Weights & Biases are paid alternatives. Selection matters less than using the system consistently and adapting it for fine-tuning rather than assuming every run trains a model from scratch. section 3.4.2

Figure 3.7 A high level view of an MLOps infrastructure with LLMs in mind. This diagram attempts to cover the full picture and the complexity of the many tools involved to make ML models work in production.

Use the infrastructure view to locate experiment tracking among the surrounding MLOps, DataOps, DevOps, and LLM production components. section 3.4

How LLMs in Production covers it

How deep it goes

Explains purpose, collaboration, fine-tuning lineage, evaluation tooling, and registry integration, but does not provide implementation steps.

Examples

Uses notebooks, MLflow, CometML, Weights & Biases, checkpoints, training experiments, and evaluation metrics as concrete examples.

What it assumes

Assumes familiarity with machine-learning experiments and notebooks while clarifying their transition into shared MLOps infrastructure.

What you can do after

Diagnose production failures, compare experiments, preserve lineage, and reduce repeated mistakes through searchable records.

How it connects

Connects experiment tracking to MLOps, model registries, feature stores, evaluation metrics, monitoring, and model fine-tuning.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning Platform EngineeringAn experiment tracker is a software component that records machine-learning experiment configurations, results, and artifacts.
• 

  liveBookmanning.com
  Machine Learning System DesignThe process of organizing, tracking, preserving, and overseeing experiments and their results.
• 

  liveBookmanning.com
  Transformers in ActionExperiment tracking is the systematic recording and organization of configurations, metrics, artifacts, and results from computational experiments.
• 

  liveBookmanning.com
  Delivering Machine Learning Projects: From design to deploymentExperiment tracking is the systematic recording of machine learning experiment parameters, inputs, outputs, and performance results.

Compare how 5 books explain experiment tracking →

Related concepts

• MLflowuses · MLflow is an open-source platform for managing the machine-learning lifecycle, including experiment tracking, model and artifact management, and model deployment.
• Weights and Biasesuses · Weights & Biases is a machine-learning development platform for tracking experiments and managing runs, metrics, artifacts, models, and related workflows.
• mlopsused by · MLOps is the discipline of applying engineering and operational practices to develop, deploy, monitor, maintain, and manage machine-learning systems throughout their life cycle.
• model registryused by · A model registry is a system for storing, organizing, versioning, and managing machine-learning models and their associated metadata throughout their lifecycle.
• performanceuses · Performance is the measurable effectiveness or operational behavior of a system under specified conditions, assessed through relevant qualities such as task results, speed, responsiveness, throughput, or resource use.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.