A prototype can return generated text alone, but a production model service also needs token counts to attribute costs. Token usage records the prompt tokens sent to a model, the completion tokens generated, and their total. Capture these counts at the source: reconstructing them later from logs or estimates risks reconciliation errors. section 3.1.1 section 3.8.1
Listing 3.1 Basic chat request and response structure
Request:
model: "gpt-4o" #A
messages: [ #B
{role: "system", content: "You are a helpful patient intake
➥assistant..."},
{role: "user", content: "What documents do I need for my appointment?"}
]
Response:
content: "Please bring your insurance card, a photo ID,
➥and any referral paperwork..."
model: "gpt-4o"
usage: {prompt_tokens: 847,
➥completion_tokens: 156} #C
A chat response carries prompt and completion token counts alongside the generated content and model name. section 3.1.1
Record usage with the provider and model that actually handled the request, not just the requested model: a fallback may have redirected it. Include workflow and user identifiers when available so costs can be traced to features and consumers. A custom endpoint can be tracked this way if it reports token usage. section 3.8.1 section 3.1.4
Listing 3.16 RequestMetrics dataclass for observability
@dataclass
class RequestMetrics:
timestamp: datetime
provider: str #A
model: str
prompt_tokens: int
completion_tokens: int
total_tokens: int
duration_ms: float
time_to_first_token_ms: Optional[float] #B
success: bool
error_type: Optional[str]
cache_hit: bool
workflow_id: Optional[str] #C
user_id: Optional[str] #C
The request metrics structure groups token counts with model identity, cache status, timing, and attribution fields. section 3.8.1
Token counts feed cost calculations, but cost also depends on the provider and model, as well as caching. Track tokens served from a response cache separately from tokens benefiting from prompt caching: the former have zero cost, while the latter have reduced cost. section 3.8.1
Aggregate request metrics by workflow, model, provider, user, and time to find spending drivers and trends. A gradual rise in tokens per request can signal prompt bloat. Compare cost with quality before switching models: lower token spending alone does not establish that a workflow is producing useful results. section 3.8.2 section 3.8.3 section 7.1.3





