liveBook · Manning

Tokenizer

As explained in Data Analysis with LLMs

A tokenizer is a software component that transforms text into the token identifiers used by a language model.

A tokenizer transforms text into the token IDs used by a language model. Those IDs let an API target particular tokens when configuring generation behavior. section 9.3

Finding class-label token IDs

For a GPT-3.5 classifier, the tokenizer identifies pos as token ID 981 and neg as token ID 29875. These IDs can then be mapped to logit-bias values, such as +100, to increase the labels' selection probability. section 9.3

Token IDs alone do not guarantee correct classification: unrestricted biased generation can repeat or combine labels. Limiting output to one token restores the demonstrated classifier behavior.

How Data Analysis with LLMs covers it

How deep it goes
Explains the transformation and demonstrates ID lookup, but stops short of tokenizer implementation internals.
Examples
Demonstrates GPT-3.5 class labels pos and neg, IDs 981 and 29875, and biased API requests.
What it assumes
Assumes familiarity with language-model API calls and token IDs while teaching how to retrieve IDs from text.
What you can do after
Operate the GPT tokenizer interface to find token IDs and configure targeted model generation.
How it connects
Connects tokenizers with token IDs, language models, logit bias, and classification.
Cover of Data Analysis with LLMs
liveBookmanning.com
Read Data Analysis with LLMs on liveBook Get the book at manning.com

Also explained in

Compare how 16 books explain tokenizer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Analysis with LLMs. The section links point to the parts of the book it draws on.