English-to-French translation maps an English sentence to an equivalent French sentence. A Transformer performs this as sequence generation: it reads the English input and predicts French tokens one at a time, using the translation prefix and encoder output. section 9.0 section 10.3
The workflow combines a parallel dataset, multilingual subword tokenization, vocabulary indexes, and Transformer training. Compared with a custom word-level tokenizer, the pretrained XLM tokenizer can split unfamiliar words into smaller pieces such as “fr” and “ench.” section 10.1.1 section 9.0
A practical English-to-French pipeline proceeds as follows:
- Load the en2fr.csv file, containing 47,173 English–French sentence pairs. section 10.1.1
- Tokenize both languages with the pretrained XLM tokenizer, adding BOS and EOS markers around each sentence. section 10.1.1
- Count frequent tokens, assign indexes, and reserve PAD and UNK indexes for sequence handling and unknown tokens. section 10.1.1
- Train the Transformer in batches to predict the next French token, then generate French translations for new English phrases. section 10.3
A small translation pair
The English phrase “How are you?” corresponds to “Comment êtes-vous?”. Tokenization represents the pair as English tokens such as “how</w>” and French tokens such as “comment</w>”, with punctuation retained as a token. section 10.1.1
Listing 10.1 A pretrained tokenizer
from transformers import XLMTokenizer #1
tokenizer = XLMTokenizer.from_pretrained("xlm-clm-enfr-1024")
tokenized_en=tokenizer.tokenize("I don't speak French.") #2
print(tokenized_en)
tokenized_fr=tokenizer.tokenize("Je ne parle pas français.") #3
print(tokenized_fr)
print(tokenizer.tokenize("How are you?"))
print(tokenizer.tokenize("Comment êtes-vous?"))
Inspect the pretrained XLM tokenizer on English and French phrases, including subword splits and punctuation. section 10.1.1
