The Common Crawl corpus is a large open-access collection of web data used as a primary training source for many large language models. section 3.2 section 3.3
Its apparent multilingual breadth masks a strong imbalance: more than 90% of its linguistic content comes from ten languages, with English making up nearly half. section 3.3
This imbalance creates several consequences:
- Models trained on the corpus may perform less reliably for languages using different scripts, grammatical structures, or cultural contexts. section 3.3
- Filtering portions of the corpus for hate speech or explicit content can unintentionally suppress marginalized voices, including LGBTQ+ communities. section 3.2
- Static web data can also reflect changing social movements unevenly, while frequent retraining remains costly and resource-intensive. section 3.2
Representation is not the same as coverage
A language may occupy a visible share of training data without receiving equivalent cultural or linguistic depth. The corpus’s concentration in a small number of languages illustrates why numerical inclusion does not guarantee comparable model performance. section 3.3
