liveBook · Manning

Tokenization

As explained in Rearchitecting LLMs

Tokenization is the process of converting text into discrete numerical units that a language model can process.

Tokenization converts raw text into numerical token IDs a language model can process. In training, it produces tensors such as input_ids, attention_mask, and labels. section 2.5

A training tokenization stage can:

  • Pad or truncate text to a chosen maximum length, such as 128 tokens. section 2.5
  • Copy input IDs into labels for the training dataset. section 2.5
  • Check that paired prompts have equal token counts so corresponding positions can be compared. section 11.1.1
def tokenize_for_kd(examples, max_length=128):
    texts = [ex["text"] for ex in examples]
    tokenized = tokenizer(
        texts,
        padding="max_length",
        truncation=True,
        max_length=max_length,
        return_tensors="pt"
    )
    return {
        "input_ids": tokenized["input_ids"],
        "attention_mask": tokenized["attention_mask"],
        "labels": tokenized["input_ids"].clone()
    }
 
# Process the raw text samples
recovery_samples = list(distillation_dataset) #1
tokenized_data = tokenize_for_kd(recovery_samples)

Example function that tokenizes text with padding and truncation, then returns input IDs, attention masks, and labels. section 2.5

How Rearchitecting LLMs covers it

How deep it goes
Defines token IDs and tensor shapes, then implements padding, truncation, attention masks, and labels without detailing tokenization algorithms.
Examples
Uses SlimPajama samples with 128-token padded or truncated inputs and paired prompts checked for equal token counts.
What you can do after
Apply it via a Hugging Face tokenizer to create input IDs, attention masks, and labels for training.
How it connects
Connects tokenization to attention masks, PyTorch datasets, DataLoaders, and knowledge distillation.

Also explained in

Compare how 36 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.