XLM is a pretrained multilingual language model used here as a tokenizer for English–French translation. Its subword tokenization handles multiple languages more effectively than a custom word-level tokenizer. section 10.1.1
XLM tokenization provides three practical behaviors:
- Splits ordinary words and punctuation into tokens, such as “I don't speak French.” into English subword tokens. section 10.1.1
- Divides unfamiliar or longer words into subwords, such as “French” into “fr” and “ench.” section 10.1.1
- Marks word boundaries with </w>, except between subwords belonging to the same word. section 10.1.1
Tokenizing English and French
Load XLMTokenizer with the pretrained model “xlm-clm-enfr-1024”, then tokenize “I don't speak French.” and “Je ne parle pas français.”. The resulting tokens include ['i</w>', 'don</w>', "'t</w>", 'speak</w>', 'fr', 'ench</w>'] and ['je</w>', 'ne</w>', 'parle</w>', 'pas</w>', 'franc', 'ais</w>'].
