Tokenization converts text into discrete pieces—words, word parts, punctuation, or symbols—that a language model can process. The model maps each token to a numerical index in its vocabulary, predicts probabilities for the next index, and then generates the corresponding token. section 2.1.1 section 2.1.2
Use this mental model when estimating what an LLM will consume:
- Text is not measured reliably by characters or words; token boundaries depend on the model’s vocabulary encoding. section 2.1.2
- Formatting counts too: braces, quotation marks, field names, and other JSON overhead become tokens. section 2.1.2
- Token counts affect context limits, uncertainty, and cost, with input and output tokens priced separately. section 2.1.2 section 2.6
- Measure actual counts with tiktoken or Agents SDK telemetry instead of estimating from visible text length. section 2.1.2 section 2.6
Plain text versus JSON
Equivalent content can consume different numbers of tokens. One comparison tokenizes regular text into six tokens but JSON into thirteen because punctuation, quotes, braces, and field names are tokenized as well. JSON remains usable, but its structure adds token overhead. section 2.1.2
Figure 2.1 The training cycle and inference/generation process of a large language model. On the left is the model’s training process, in which documents are ingested to train the LLM in a first pass. On the right, a user enters a prompt, which is first tokenized and fed into the model. The model then outputs probabilities it uses to sample and produce the next token.
Trace how documents become training inputs and how a user prompt becomes tokenized input, next-token probabilities, and generated output. section 2.1.1
Figure 2.2 A comparison of tokenization in regular text and JSON
Compare how regular text and JSON are divided into tokens, highlighting why equivalent content can have different token costs. section 2.1.2
