Duplicate detection finds and removes repeated or substantially similar content from a dataset. In language-model training, the same quote may appear hundreds of times across forums and blogs, causing memorization instead of meaningful generalization. section 11.7
Removing duplicates helps to:
- Prevent the model from assigning excessive probability to repeated phrases. section 11.7
- Encourage learning general meaning rather than reproducing memorized text verbatim. section 11.7
A repeated quote
Imagine one quotation copied into hundreds of forum posts and blogs. Without duplicate removal, that repetition makes the quote seem unusually important; the model may reproduce it even when it is an incorrect response. section 11.7
