liveBook · Manning

Quality filter

As explained in LLM Customization and Fine-Tuning

A screening process that removes low-quality examples before they are used for training.

A quality filter screens training examples before they reach a model. In knowledge distillation, it removes teacher responses that could teach a student terse answers or repetitive filler. The tradeoff is direct: an aggressive filter discards useful data, while a lenient one lets poor examples through. section 7.1.6 section 7.7.2

A basic filter checks response length and degenerate repetition. Test it with a normal response, responses that are too short and too long, and a repetitive response. Include these tests in CI and update them when domain-specific thresholds change. section 7.1.6 section 7.7.2

Calibrate against the teacher and domain rather than treating a rejection rate as a universal target:

  • Start by examining a small batch. Dropping 10%–25% of outputs is a workable starting point for the described frontier-teacher setup, but a clean local teacher lost only 1 of 200 examples. section 7.1.6
  • After each rule change, manually review 20 rejected outputs. Loosen the filter if most are reasonable answers; a high rejection rate can be appropriate if the rejects are mostly incomplete, repetitive, or refusals on benign prompts. section 7.1.6
  • Use domain-expert spot checks and adjust thresholds during refinement, then retrain and compare with the baseline. section 7.6.6

Clinical training data

A clinical fine-tuning dataset used deduplication, length filtering, and manual review of a 200-example stratified sample. Filtering removed about 8% of the initial data, mainly near-duplicate questions and examples with incomplete reasoning or reasoning that contradicted the final answer. Early unfiltered experiments sometimes produced confident but internally contradictory reasoning. section 6.12.1 section 6.12.6

Length and repetition checks do not establish safety or appropriateness: a harmful answer can be fluent and the right length. Add safety-aware spot checks and evaluation. Do not indiscriminately remove refusals; refusals paired with harmful prompts can teach a student when to decline. section 7.8.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Details length and repetition checks, rejection-rate calibration, and manual review, while distinguishing these checks from safety assessment.
Examples
Uses a teacher-output run that removed one example and a clinical dataset that lost about 8% during filtering.
What you can do after
Operate and configure a filter by testing edge cases, reviewing rejects, and adjusting thresholds for the domain.
How it connects
Connects knowledge distillation and supervised fine-tuning with deduplication and safety evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.