A data-quality audit checks whether collected and normalized data is clean, balanced, consistent, and suitable for model training. It turns vague concerns into measurable targets. section 11.7
Useful checks include:
- Duplicate rate, to limit memorization and unnecessary dataset size. section 11.7
- Sentence length, toxicity, profanity, language purity, and privacy leaks. section 11.7
- Non-UTF8 characters and formatting problems that can disrupt processing. section 11.7
From raw text to measurable quality
An Alice’s Adventures in Wonderland excerpt can be stripped of boilerplate, normalized into consistent characters and whitespace, then checked against targets such as fewer than 5% exact-paragraph duplicates and zero detected emails, phone numbers, or ID numbers. section 11.7
Table 11.2 A summary of areas important for data quality
|
Metric |
Why it matters |
Goals |
| Duplicate rate |
It prevents memorization and reduces dataset size. |
< 5% exact-paragraph matches |
| Average sentence length |
Very short sentences lack context, and very long sentences overflow the model window. |
5–40 tokens per segmented sentence |
| Toxicity rate |
It keeps generation safe and inoffensive. |
< 0.3% words flagged for toxicity |
| Profanity ratio |
It provides a more precise count than toxicity rate. |
Depends on the use case; can be < 0.2% for general models but can be higher for “edgy” domains |
| Language purity |
Mixed languages confuse the model and add unnecessary vocabulary. |
> 98% of the content in the desired language |
| PII leaks |
It reduces privacy and legal risks. |
0 emails, 0 phone numbers, 0 ID numbers |
| Non-UTF8 character rate |
Binary junk characters can cause the tokenizer (explored next) to crash. |
0 control characters (realistically, < 0.01% is good enough) |
| Forbidden phrases |
It prevents generation of phrases you don’t want to see. |
0 occurrences of forbidden phrases |
| Markup density |
Markup like HTML and markdown noise wastes the context window and makes tokenizing less accurate. |
< 2% lines containing code characters such as <, [, ^, >, ], +, > and links |
| Readability score |
It makes the LLM generate text that’s easy to read (written for a certain reading level). |
The Flesch-Kincaid Grade-Level readability score (F-K score) is a formula to determine ease of readability. A score of 6 means 11-year-olds can read it easily; a score of 12 means 17-year-olds can read it easily. |
| Domain balance |
If you curate multiple domains (medical, fiction, geography, and so on), you want to represent them as intended. |
No more than 10% variance per domain. If you planned 20% medical content, for example, your actual data shouldn’t be less than 10% or more than 30%. |
Use the quality checklist to compare metrics, reasons, and target thresholds. section 11.7
