liveBook · Manning

Spot-checking

As explained in LLM Customization and Fine-Tuning

Human review of a sample of examples to identify quality problems that automated checks may miss.

Spot-checking uses human review of a sample to catch quality problems automated checks miss, including factual errors, inconsistent reasoning, and technically correct but unhelpful answers. section 7.1.3

Use spot-checking as a quality gate:

  • After automated filtering, have a domain expert review a random 10% sample and record the error rate as a data-quality metric. section 7.1.3
  • If more than 20% of the sample has quality issues, adjust the prompts or teacher configuration before proceeding. section 7.1.3
  • When expert review is unavailable, use a different model to review outputs as a lower-cost supplement, not a replacement for human review in the initial iteration. section 7.1.3

Before shipping a DPO model

Generate 20 to 30 responses to representative production prompts and have a domain expert review them. Metrics show aggregate trends; spot checks reveal individual failure modes. section 8.2.9

How LLM Customization and Fine-Tuning covers it

How deep it goes
Specifies sample sizes, a quality threshold, and follow-up actions, but does not prescribe a detailed reviewer rubric.
Examples
Reviews a random 10% of filtered teacher outputs and 20 to 30 responses to representative production prompts.
What you can do after
Evaluate output quality, track sample error rates, and decide when to revise generation settings or block deployment.
How it connects
Connects automated filtering and knowledge distillation with LLM-as-a-judge review and DPO evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.