liveBook · Manning

Canary check

As explained in LLM Customization and Fine-Tuning

A check of model behavior on selected prompts to detect changes or regressions in its responses.

A canary check evaluates model behavior on selected prompts using Token-F1. Unlike a vocabulary-drift detector, which signals changes in incoming prompts, a canary can reveal when answers have gone stale because the world changed—even when model weights remain static. section 9.2.3

Treat the canary as a confirming signal: a red drift score triggers a canary run before any retraining decision; a low score calls for monitoring only. section 9.2.3

Figure 9.3 Responding to a drift signal: a red drift score triggers a canary run before any retrain decision.

Decision flow connecting a red drift signal to a canary run before retraining. section 9.2.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Measures behavior with Token-F1 and distinguishes behavioral changes from vocabulary distance, without specifying prompt selection or scoring thresholds.
Examples
Evaluates selected canary prompts with Token-F1 as a confirming signal after a red drift score.
What you can do after
Diagnose whether model answers have gone stale before deciding to retrain.
How it connects
Connects distribution drift and concept drift to model retraining decisions.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.