liveBook · Manning

Automated filter

As explained in LLM Customization and Fine-Tuning

The use of programmatic checks to identify and remove examples that fail specified quality criteria.

Automated filters remove teacher-generated distillation examples that fail basic checks before student training. They catch common mechanical failures, not whether an answer is factually sound or useful. section 7.1.3 section 7.1

Common checks target:

Listing 7.2 Preparing distillation data
# Key sections of chapter07/scripts/prepare_distillation_data.py
 
def quality_filter(example: dict, min_words: int, max_words: int) -> bool:
    """Basic quality filter on teacher responses."""
    msgs = example["messages"]
    assistant_msg = next(
        (m["content"] for m in msgs if m["role"] == "assistant"), ""
    )
    word_count = len(assistant_msg.split())
 
    if word_count < min_words:                                 #A
        return False
    if word_count > max_words:                                 #B
        return False
    # Check for degenerate repetition
    sentences = assistant_msg.split(".")
    if len(sentences) > 3:
        unique_sentences = {s.strip().lower() for s in sentences if s.strip()}
        if len(unique_sentences) < len(sentences) * 0.5:      #C
            return False
    return True
 
# In main():
filtered = [
    ex for ex in examples
    if quality_filter(ex, args.min_response_words, args.max_response_words)
]
# ... shuffle, split, write manifest

This Python filter applies word-count bounds and rejects responses with excessive repeated sentences. section 7.1.3

Spot-check threshold

Review a random 10% sample. If more than 20% of that sample has quality issues, adjust the prompts or teacher configuration before proceeding. section 7.1.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Checks length, repetition, and format; leaves subtle factual inaccuracies, inconsistent reasoning, and unhelpful answers to manual review.
Examples
Uses word-count bounds and repeated-sentence detection, with one reported run removing 1 of 200 outputs.
What you can do after
Operate/configure response-length and repetition checks, then send filtered data to manual spot-checking.
How it connects
Links quality filtering to knowledge distillation, supervised fine-tuning, and LLM-as-a-judge review.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.