liveBook · Manning

Token count

As explained in Designing AI Systems

Token usage is the quantity of language-model input and output tokens consumed by an AI operation.

A prototype can return generated text alone, but a production model service also needs token counts to attribute costs. Token usage records the prompt tokens sent to a model, the completion tokens generated, and their total. Capture these counts at the source: reconstructing them later from logs or estimates risks reconciliation errors. section 3.1.1 section 3.8.1

Listing 3.1 Basic chat request and response structure
Request:
  model: "gpt-4o"    #A
  messages: [    #B
    {role: "system", content: "You are a helpful patient intake 
➥assistant..."},
    {role: "user", content: "What documents do I need for my appointment?"}
  ]
 
Response:
  content: "Please bring your insurance card, a photo ID, 
➥and any referral paperwork..."
  model: "gpt-4o"
  usage: {prompt_tokens: 847, 
➥completion_tokens: 156}    #C
 

A chat response carries prompt and completion token counts alongside the generated content and model name. section 3.1.1

Record usage with the provider and model that actually handled the request, not just the requested model: a fallback may have redirected it. Include workflow and user identifiers when available so costs can be traced to features and consumers. A custom endpoint can be tracked this way if it reports token usage. section 3.8.1 section 3.1.4

Listing 3.16 RequestMetrics dataclass for observability
@dataclass
class RequestMetrics:
    timestamp: datetime
    provider: str    #A
    model: str
    prompt_tokens: int
    completion_tokens: int
    total_tokens: int
    duration_ms: float
    time_to_first_token_ms: Optional[float]    #B
    success: bool
    error_type: Optional[str]
    cache_hit: bool
    workflow_id: Optional[str]    #C
    user_id: Optional[str]    #C
 

The request metrics structure groups token counts with model identity, cache status, timing, and attribution fields. section 3.8.1

Token counts feed cost calculations, but cost also depends on the provider and model, as well as caching. Track tokens served from a response cache separately from tokens benefiting from prompt caching: the former have zero cost, while the latter have reduced cost. section 3.8.1

Aggregate request metrics by workflow, model, provider, user, and time to find spending drivers and trends. A gradual rise in tokens per request can signal prompt bloat. Compare cost with quality before switching models: lower token spending alone does not establish that a workflow is producing useful results. section 3.8.2 section 3.8.3 section 7.1.3

How Designing AI Systems covers it

How deep it goes
Defines prompt, completion, total, and cache-related metrics and specifies request-level recording and aggregation.
Examples
Shows a chat response with 847 prompt tokens and 156 completion tokens and a request-metrics dataclass.
What you can do after
Diagnose spending drivers and prompt bloat using token metrics grouped by workflow, model, user, and time.
How it connects
Connects token usage with observability, caching, model fallback, cost attribution, and quality scores.

Also explained in

Compare how 6 books explain token count →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.