Token

In the context of large language models (LLMs) and natural language processing (NLP), a "token" is a fundamental unit of text that is used for processing by machine learning models. Tokens can represent characters, words, or subwords, depending on the tokenization strategy employed. The transformation of text into tokens is a critical step in preparing data for model training and inference, as it allows text to be converted into numerical representations that can be processed by algorithms.

Overview of Tokens

Tokens are the basic units of text that are transformed into numerical representations for processing in machine learning models. Each token is mapped to a unique integer, known as a token ID, through a vocabulary. This transformation is essential for converting text data into a format that can be utilized by machine learning algorithms. Source

Vocabulary Creation

The process of creating a vocabulary involves tokenizing the entire text within a training dataset into individual tokens. These tokens are then sorted alphabetically, and any duplicates are removed. The resulting unique tokens are compiled into a vocabulary, which establishes a mapping from each unique token to a unique integer value. This process is illustrated in Figure 2.6.

Figure 2.6 We build a vocabulary by tokenizing the entire text in a training dataset into individual tokens. These individual tokens are then sorted alphabetically, and duplicate tokens are removed. The unique tokens are then aggregated into a vocabulary that defines a mapping from each unique token to a unique integer value. The depicted vocabulary is purposefully small and contains no punctuation or special characters for simplicity.

Example: Creating a Vocabulary

The following code snippet demonstrates how to create a vocabulary by enumerating over a list of words and assigning each word a unique integer ID:

vocab = {token:integer for integer,token in enumerate(all_words)}
for i, item in enumerate(vocab.items()):
    print(item)
    if i >= 50:
        break

Listing 2.2 Creating a vocabulary

Converting Tokens to Token IDs

Once a vocabulary is established, each token can be converted to its corresponding token ID. This conversion is a crucial intermediate step before transforming token IDs into embedding vectors, which are used for further processing in machine learning models. Source

Token Usage in Models

Tokens play a significant role in models like GPT (Generative Pre-trained Transformer), where they are used as inputs and outputs. Figure 6.11 illustrates a GPT model with a four-token example input and output. The model's output tensor consists of two columns due to a modified output layer, and the focus is on the last row corresponding to the last token when fine-tuning the model for tasks such as spam classification.

Figure 6.11 The GPT model with a four-token example input and output. The output tensor consists of two columns due to the modified output layer. We are only interested in the last row corresponding to the last token when fine-tuning the model for spam classification.

Fine-Tuning Tokens

In the context of fine-tuning, focusing on the last output token is often preferred. This is because the last token accumulates the most information due to the causal attention mask, which allows it to attend to all preceding tokens. Fine-tuning the first token, on the other hand, may lead to different predictive performance. Source

Tokenization Process

Tokenization is the process of converting a sequence of text into tokens. This process involves several steps, including normalization, pre-tokenization, model application, and post-processing. Each step plays a crucial role in preparing the text data for model training and inference. Source

Example: Training a Tokenizer

To illustrate the tokenization process, consider the following example where we train a tokenizer using the text of the "Odyssey," a classic epic poem by Homer. We utilize the tokenizers library, which provides a flexible and efficient way to handle tokenization.

from tokenizers import Tokenizer
from tokenizers.models import WordLevel
from tokenizers.pre_tokenizers import Whitespace
from tokenizers.trainers import WordLevelTrainer

tokenizer = Tokenizer(WordLevel(unk_token="[UNK]"))  #1
tokenizer.pre_tokenizer = Whitespace()  #2
trainer = WordLevelTrainer(special_tokens=["[UNK]"])  #3
tokenizer.train(["../data/p2ch9/odyssey.txt"], trainer=trainer)

vocab_size = tokenizer.get_vocab_size()
decode = tokenizer.decode
encode = tokenizer.encode

1. Model Selection: We use the WordLevel model from the tokenizers library. This model is designed to train a tokenizer on a word-level vocabulary. A special token [UNK] is specified to represent unknown tokens.

2. Pre-tokenization: The tokenizer is set to split the text at any whitespace before any tokenization is done. This step is crucial for segmenting the text into manageable units.

3. Training: The WordLevelTrainer is used to specify the core algorithm for tokenization. While this example uses a straightforward word-level approach, other options like Byte-Pair-Encoding and Wordpiece are available for more complex tokenization needs.

After training, the tokenizer can provide the vocabulary size and functions to encode and decode text sequences. This setup allows for the construction of datasets necessary for training language models by processing text into token sequences and their corresponding targets. Source

| Book Title | Usage of Token | Technical Depth | Connections to Other Concepts | Examples Used | Practical Application |
|------------|----------------|-----------------|-------------------------------|---------------|-----------------------|
| [Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch) | Discusses tokens as fundamental units of text transformed into numerical representations for machine learning models. | Explains vocabulary creation, token ID conversion, and their role in models like GPT. | Connects tokens to vocabulary creation, embedding vectors, and model outputs. | Provides code snippets for creating a vocabulary and illustrates token usage in GPT models. | Focuses on fine-tuning models using tokens, especially the last token for tasks like spam classification. [more](https://livebook.manning.com/book/build-a-large-language-model-from-scratch/chapter-2/63) |
| [Deep Learning with PyTorch, Second Edition](https://www.manning.com/books/deep-learning-with-pytorch-second-edition) | Defines tokens as fundamental units in NLP and deep learning, forming sequences for input and output processing. | Details the tokenization process including normalization, pre-tokenization, and model application. | Discusses the impact of tokenization strategy on model performance and efficiency. | Uses the `tokenizers` library to train a tokenizer on the text of the "Odyssey" by Homer. | Provides practical steps for training a tokenizer and constructing datasets for language models. [more](https://livebook.manning.com/book/deep-learning-with-pytorch-second-edition/chapter-9/186) |