Token

As explained in AI Agents in Action, Second Edition

A token is a discrete unit of text represented numerically for processing by a language model.

A token is a word or word part represented by a numerical index in a language model’s vocabulary. Tokenization breaks text into these units; the model predicts possible next-token indices, and a sampling step selects one to generate. section 2.1.1 section 2.1.2

Text length does not reliably indicate token count. Punctuation, quotes, and field names in JSON can increase the count, affecting the cost of sending data to a model. section 2.1.2

JSON and regular text

A JSON block and equivalent regular text are shown as 13 and six tokens, respectively. Measure text with tiktoken or use token counts reported by the OpenAI Agents SDK; providers price input and output tokens separately. section 2.1.2 section 2.6

Figure 2.2 A comparison of tokenization in regular text and JSON

Compare how regular text and JSON are split into tokens. section 2.1.2

How AI Agents in Action, Second Edition covers it

How deep it goes

Defines text-to-index tokenization and connects next-token prediction to sampling.

Examples

Compares JSON at 13 tokens with equivalent regular text at six tokens.

What you can do after

Apply tiktoken or SDK telemetry to measure token counts and manage budgets and context windows.

How it connects

Links tokens to tokenization, model vocabularies, probability distributions, sampling, and prompting.

liveBookmanning.com

Read AI Agents in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  How Large Language Models WorkA token is a discrete symbolic unit that represents a subword through a unique numeric identifier for processing by a language model.
• 

  liveBookmanning.com
  Build Python Web Apps with StreamlitA token is a chunk of text, such as a word or word fragment, processed as an input or generated as an output by a language model.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA token is a discrete unit of text processed by a language model, such as a word, subword, character, or punctuation symbol.
• 

  liveBookmanning.com
  Grokking AI ApplicationsA token is a discrete text unit that a language model converts into numerical representations and processes as input or output.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A token is a discrete unit produced by segmenting text, such as a word, punctuation mark, or subword.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA discrete lexical unit obtained by splitting source text into identifiable pieces, including individual characters in character-based processing.
• 

  liveBookmanning.com
  Sutskever's ListA discrete unit of text processed by a language model, which may represent a word or a word fragment.
• 

  liveBookmanning.com
  Software Security for DevelopersA token is a security artifact carrying authorization or identity information between participants in an authentication or authorization protocol.
• 

  liveBookmanning.com
  Generative AI in ActionA token is a discrete unit into which text or other model input is encoded for processing by a language model.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA discrete unit in a text sequence, such as a word or subword, that serves as the basic item attended over by the model.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA discrete text unit produced by tokenization, such as a word, punctuation mark, or special character, used as an element of a sequence.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA token is a discrete unit of text, such as a word, subword, or symbol, used as an input element for language-model processing.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsA token is a discrete representation of a sequence segment, including a fixed-length set of past time steps and associated features.
• 

  liveBookmanning.com
  The AI Pocket BookA discrete text unit fed to a language model as input, which may correspond to a subword or word rather than an individual letter.
• 

  liveBookmanning.com
  MongoDB 8.0 in Action, Third EditionA token is a text unit produced during analysis for indexing and searching, such as a word, phrase, or symbol.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A discrete unit of text that a language model processes, roughly corresponding to words or punctuation marks in this section.
• 

  liveBookmanning.com
  Starting Data Analytics with Generative AI and PythonA token is a unit of text processed by a language model, such as a word, word fragment, or punctuation mark.
• 

  liveBookmanning.com
  Build AI-Enhanced Web AppsA token is a discrete unit of text, such as a word, subword, or character, processed by a language model.
• 

  liveBookmanning.com
  Data Analysis with LLMsA token is the atomic unit of text representation processed by a language model, typically corresponding to about four characters.
• 

  liveBookmanning.com
  Learn Rust in a Month of LunchesA token is a lexical unit of source input, such as a number, identifier, punctuation mark, or other syntactic element.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA text unit produced by tokenization, which may be a character, word, or subword depending on the tokenizer design.
• 

  liveBookmanning.com
  Hugging Face in ActionA token is a unit of text that a machine learning model processes as a single input element.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionA token is a word or subword unit into which a text sequence is divided for computational processing.
• 

  liveBookmanning.com
  Kotlin in Action, 2EA token is a lexical unit representing either a syntactic symbol or a value in an input language.
• 

  liveBookmanning.com
  Just Use Postgres!A token is a smaller text component produced by parsing a document, such as a word, number, or phrase.
• 

  liveBookmanning.com
  LLMs in ProductionA token is a discrete unit of text representation processed by a language model.
• 

  liveBookmanning.com
  Go in Action, Second EditionA token is a piece of input data produced by a scanner according to its splitting rules.
• 

  liveBookmanning.com
  Inside AIA token is a sequence of characters, word fragment, or related text unit processed by a language model.
• 

  liveBookmanning.com
  Causal AIA token is a sequence of characters treated as one unit of input or output by a language model.
• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)A discrete text unit used as the basic element of language-model generation.

Compare how 31 books explain token →

Related concepts

• tokenizationproduced by · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
• language modelproduced by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• context windowmeasured by · A context window is the bounded sequence of tokens or other input information a model can access when processing or generating an output.
• Stable Diffusionused by · Stable Diffusion is a family of latent diffusion models that generates and edits images from text and other conditioning inputs.
• token streamingused by · Token streaming is the incremental delivery of generated language-model output as tokens become available, rather than waiting for the complete output.

This explainer was generated by Manning's knowledge graph from the text of AI Agents in Action, Second Edition. The section links point to the parts of the book it draws on.