Personal data is information that can identify or be linked to a specific person, including phone numbers and credit-card numbers. In language-model training, removing it helps prevent privacy and legal problems and reduces the chance that a model reproduces sensitive information. section 11.7
A data-cleaning pass treats personal data as a safety and compliance concern:
- Find sensitive details in the training data. section 11.7
- Strip phone numbers, credit-card numbers, and other personally identifying information before training. section 11.7
- Check for remaining email addresses, phone numbers, and ID numbers; the stated quality goal is zero leaks. section 11.7
Protecting training data
Suppose a collected document contains a customer’s phone number and credit-card number. A preprocessing filter removes those details before the document enters the training set, reducing the risk that the model memorizes and reproduces them. section 11.7
Table 11.2 A summary of areas important for data quality
|
Metric |
Why it matters |
Goals |
| Duplicate rate |
It prevents memorization and reduces dataset size. |
< 5% exact-paragraph matches |
| Average sentence length |
Very short sentences lack context, and very long sentences overflow the model window. |
5–40 tokens per segmented sentence |
| Toxicity rate |
It keeps generation safe and inoffensive. |
< 0.3% words flagged for toxicity |
| Profanity ratio |
It provides a more precise count than toxicity rate. |
Depends on the use case; can be < 0.2% for general models but can be higher for “edgy” domains |
| Language purity |
Mixed languages confuse the model and add unnecessary vocabulary. |
> 98% of the content in the desired language |
| PII leaks |
It reduces privacy and legal risks. |
0 emails, 0 phone numbers, 0 ID numbers |
| Non-UTF8 character rate |
Binary junk characters can cause the tokenizer (explored next) to crash. |
0 control characters (realistically, < 0.01% is good enough) |
| Forbidden phrases |
It prevents generation of phrases you don’t want to see. |
0 occurrences of forbidden phrases |
| Markup density |
Markup like HTML and markdown noise wastes the context window and makes tokenizing less accurate. |
< 2% lines containing code characters such as <, [, ^, >, ], +, > and links |
| Readability score |
It makes the LLM generate text that’s easy to read (written for a certain reading level). |
The Flesch-Kincaid Grade-Level readability score (F-K score) is a formula to determine ease of readability. A score of 6 means 11-year-olds can read it easily; a score of 12 means 17-year-olds can read it easily. |
| Domain balance |
If you curate multiple domains (medical, fiction, geography, and so on), you want to represent them as intended. |
No more than 10% variance per domain. If you planned 20% medical content, for example, your actual data shouldn’t be less than 10% or more than 30%. |
Use the data-quality table to compare PII leaks with other training-data quality measures. section 11.7
