Encoding converts text into numeric representations that neural networks can process. A character or word receives an integer or a one-hot vector identifying its position in an encoding space. section 4.5.2 section 9.3
Character encoding can use ASCII for a small English character set, while Unicode represents a much wider range through encodings such as UTF-8, UTF-16, and UTF-32. Preprocessing may lowercase text or remove irrelevant punctuation. section 4.5.2
Common neural-network representations include:
- Character-level one-hot encoding assigns each character a vector containing one 1 and otherwise 0s; the vector length equals the character-set size. section 4.5.2
- Word-level one-hot encoding builds a vocabulary-to-index mapping and assigns one vector per word, but can produce extremely wide vectors and unknown-word problems. section 4.5.3
- Integer encoding maps characters or tokens to IDs; shifted input and target sequences then support next-character prediction. section 9.3
- Subword tokenization offers an intermediate split between characters and words, while tokenizer pipelines can normalize, pretokenize, model, and postprocess text. section 4.5.3 section 9.8
Encoding a name for next-character prediction
With a character-to-index mapping, "$ada$" becomes [0, 1, 4, 1, 0]. Its shifted target is [1, 4, 1, 0], and padding can align examples into batches while -1 marks ignored target positions. section 9.3
# In[]: from tokenizers import Tokenizer from tokenizers.models import WordLevel from tokenizers.pre_tokenizers import Whitespace from tokenizers.trainers import WordLevelTrainer tokenizer = Tokenizer(WordLevel(unk_token="[UNK]")) #1 tokenizer.pre_tokenizer = Whitespace() #2 trainer = WordLevelTrainer(special_tokens=["[UNK]"]) #3 tokenizer.train(["../data/p2ch9/odyssey.txt"], trainer=trainer) vocab_size = tokenizer.get_vocab_size() decode = tokenizer.decode encode = tokenizer.encode
A runnable WordLevel tokenizer shows how to train a vocabulary, obtain its size, and expose encode and decode functions for model data preparation. section 9.8
