A token is a sequence of characters treated as one unit by a language model. It can be a single character, a whole word, punctuation, or part of a word; it is not necessarily a word. A tokenizer maps tokens to vocabulary IDs, which a Transformer processes as numbers. section 13.2.1
Listing 13.2 Viewing example tokens that an LLM operates upon
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2') #1
tokens = tokenizer.tokenize("Can LLMs reason counterfactually?") #2
print(tokens) #3
This GPT-2 example tokenizes “Can LLMs reason counterfactually?”; the output illustrates space markers, punctuation, and words split into subword tokens. section 13.2.1
GPT-2 assigns each token an integer from its 50,257-entry vocabulary. Autoregressive language models assign probabilities to token sequences, including whether a sequence ends; GPT-2 uses a special end-of-text token. But token-level structure is not the main target of causal analysis: the focus is on causal relations between the concepts expressed by tokens. section 13.2.1 section 13.2.3
Figure 13.2 X1 through X12 are a sequence of tokens. Whatever structure (causal or otherwise) exists between the tokens is of passing interest. We are interested in the causal relations between concepts described by the tokens.
The figure contrasts a sequence of tokens with the causal relations between the concepts those tokens describe. section 13.2.3
