liveBook · Manning

Token

As explained in Designing AI Systems

A token is a unit into which text is divided for processing and usage measurement by an AI model.

A token is a unit counted in model requests. Token metrics distinguish prompt tokens sent to a model, completion tokens it generates, and their total; providers use these counts for cost calculations. section 3.8.1

Request metrics can also record:

  • Tokens served from response cache, which have zero cost, and tokens benefiting from prompt caching, which have reduced cost. section 3.8.1
Listing 3.16 RequestMetrics dataclass for observability
@dataclass
class RequestMetrics:
    timestamp: datetime
    provider: str    #A
    model: str
    prompt_tokens: int
    completion_tokens: int
    total_tokens: int
    duration_ms: float
    time_to_first_token_ms: Optional[float]    #B
    success: bool
    error_type: Optional[str]
    cache_hit: bool
    workflow_id: Optional[str]    #C
    user_id: Optional[str]    #C
 

The request-metrics dataclass shows token counts alongside provider, model, caching, and workflow attribution fields. section 3.8.1

How Designing AI Systems covers it

How deep it goes
Distinguishes prompt, completion, total, and cached-token metrics, but does not explain tokenization mechanics.
Examples
Uses request-metric fields for prompt, completion, total, and cached tokens rather than numerical examples.
What you can do after
Evaluate model-request consumption and relate token counts to provider costs.
How it connects
Connects token usage to model services, response caching, prompt caching, and cost attribution.

Also explained in

Compare how 40 books explain token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.