Language filtering removes training text written in languages outside a model’s intended language set. It helps prevent mixed vocabularies and grammars from competing in the same Transformer weights. section 11.7
Use language filtering to:
- Reduce the chance that less-common languages are drowned out or that common languages are trained poorly. section 11.7
- Avoid unnecessarily expanding the vocabulary, whose importance becomes clearer during tokenization. section 11.7
Alice in Wonderland
An English-only excerpt needs no language filtering. A real-world multilingual dataset would first remove text in languages outside the intended set before further cleaning. section 11.7
Table 11.2 A summary of areas important for data quality
|
Metric |
Why it matters |
Goals |
| Duplicate rate |
It prevents memorization and reduces dataset size. |
< 5% exact-paragraph matches |
| Average sentence length |
Very short sentences lack context, and very long sentences overflow the model window. |
5–40 tokens per segmented sentence |
| Toxicity rate |
It keeps generation safe and inoffensive. |
< 0.3% words flagged for toxicity |
| Profanity ratio |
It provides a more precise count than toxicity rate. |
Depends on the use case; can be < 0.2% for general models but can be higher for “edgy” domains |
| Language purity |
Mixed languages confuse the model and add unnecessary vocabulary. |
> 98% of the content in the desired language |
| PII leaks |
It reduces privacy and legal risks. |
0 emails, 0 phone numbers, 0 ID numbers |
| Non-UTF8 character rate |
Binary junk characters can cause the tokenizer (explored next) to crash. |
0 control characters (realistically, < 0.01% is good enough) |
| Forbidden phrases |
It prevents generation of phrases you don’t want to see. |
0 occurrences of forbidden phrases |
| Markup density |
Markup like HTML and markdown noise wastes the context window and makes tokenizing less accurate. |
< 2% lines containing code characters such as <, [, ^, >, ], +, > and links |
| Readability score |
It makes the LLM generate text that’s easy to read (written for a certain reading level). |
The Flesch-Kincaid Grade-Level readability score (F-K score) is a formula to determine ease of readability. A score of 6 means 11-year-olds can read it easily; a score of 12 means 17-year-olds can read it easily. |
| Domain balance |
If you curate multiple domains (medical, fiction, geography, and so on), you want to represent them as intended. |
No more than 10% variance per domain. If you planned 20% medical content, for example, your actual data shouldn’t be less than 10% or more than 30%. |
Use the data-quality summary to compare language purity with other preparation checks; it gives a goal of more than 98% content in the desired language. section 11.7
