liveBook · Manning

Model registry

As explained in LLM Customization and Fine-Tuning

A system for storing model artifacts and metadata such as versions, training details, and evaluation records for audit and deployment control.

A failed model deployment is harder to reverse when the team cannot identify the previous checkpoint. A model registry records model versions and their training and evaluation metadata, providing an audit trail for deployment, reproducibility, and rollback. A lightweight registry can hold checkpoint paths rather than the model artifacts themselves. section 9.1.3 section 9.11

A JSON registry can track a version tag, training technique, base model, data hash, checkpoint path, evaluation metrics, and notes. Keep it in version control alongside the code so reviewers can inspect changes and scripts can parse the records. Register versions before deployment, even when there is only one model. section 9.1.2 section 9.1.3

A simple registry supports four operations:

  • Register a uniquely tagged version with its checkpoint and metadata. section 9.1.2 section 9.1.3
  • List versions and their statuses to inspect the deployment history. section 9.1.3
  • Promote a registered version to active, retiring the version it replaces. section 9.1.2
  • Roll back the most recent promotion, returning the prior version to active. section 9.1.2

Wire promotion and registry updates into the deployment script as a single step rather than relying on separate manual actions. Keep previous versions ready to serve: the registry identifies a rollback target, but recovery also depends on deployment infrastructure. section 9.1.3 section 9.11

Rollback after a safety regression

An IT support model fails a monthly bias and fairness check after a retrain. A registry rollback returns v1 to active and retires v2; recovery from alert to verified rollback takes under an hour. The team removes biased training examples, retrains v3, and validates it against the full safety suite before promotion. The retired v2 record remains available for audit. section 9.9.2

Table 9.2 Registry approaches by team scale
Approach
Best for
Tradeoffs
JSON file (this chapter)
Small teams, 1 to 10 models
Simple, version-controlled, no infrastructure
MLflow / W&B Registry
Mid-size teams, 10 to 50 models
UI, search, comparison; requires a server
Vertex AI / SageMaker
Large teams, regulated industries
Full lifecycle management; cloud lock-in
DVC / lakeFS
Data-heavy pipelines
Data versioning; steep learning curve
Git LFS
Large model artifacts
Integrated with Git; storage costs scale

Compare registry approaches by team scale and operational tradeoffs. section 9.1.2

A single JSON file suits a small team starting out: it is easy to inspect, diff, and maintain without extra infrastructure. Consider a managed registry when version counts grow beyond roughly 10–15 or multi-team access control is needed. Managed options add capabilities such as dashboards and deployment integration, but may require a server or introduce cloud lock-in. section 9.1.2

How LLM Customization and Fine-Tuning covers it

How deep it goes
Implements a JSON registry with registration, listing, promotion, and rollback, and gives guidance for migration to managed services.
Examples
Follows an IT support model through a safety-triggered rollback, data cleanup, retraining, and later promotion.
What you can do after
Operate a JSON registry to register, list, promote, and roll back model versions.
How it connects
Connects model versioning with evaluation metrics, deployment, rollback, and reproducibility.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.