Linguistic diversity describes the range of languages represented in communication and artificial intelligence systems. Language models often perform worse for languages and cultural contexts from lower-income regions, creating a feedback cycle: well-resourced languages gain stronger support and more training data, while underrepresented languages risk falling further behind. section 3.3
Efforts to improve representation include:
- BLOOM incorporates 46 natural languages and 13 programming languages. section 3.3
- Masakhane, Ghana NLP, and AI4Bharat focus on African, Ghanaian, and Indian languages. section 3.3
- XTREME and MMLU-ProX broaden multilingual evaluation, although culturally appropriate test sets remain resource-intensive. section 3.3
Targeted language support
Specialized models for Arabic and Southeast Asian languages can outperform general-purpose models on local languages, showing the value of targeted data collection and training. section 3.3
