liveBook · Manning

ChatChunk

As explained in Designing AI Systems

ChatChunk is a message structure representing one incremental fragment of a model's streamed response, including the fragment text, producing model, completion status, and optional usage information.

ChatChunk is the platform’s normalized unit for streamed model output. It carries one text fragment, the producing model, an optional completion reason, and optional token usage so downstream services can process streams without provider-specific logic. section 3.3.2

Use the fields to assemble output, attribute generation, and finalize operations:

  • token contains the arriving text fragment. It may be a word, part of a word, or punctuation; consumers generally append it as received. section 3.3.2
  • model identifies the producer for each fragment, allowing consumers to detect mid-stream fallback changes and attribute cost. section 3.3.2
  • finish_reason appears on the final chunk and records stop, length, tool_calls, content_filter, or error termination. section 3.3.2
  • usage typically appears only at completion because token counts become available after generation ends, making streaming cost tracking retrospective. section 3.3.2

Provider-normalized streaming

A provider adapter converts OpenAI delta objects, Anthropic text-stream events, or compatible self-hosted output into ChatChunk messages. The Model Service forwards them through gRPC streaming, while the API Gateway emits Server-Sent Events. A client appends each token until a final chunk supplies finish_reason and, typically, usage. section 3.3.1 section 3.3.2

Listing 3.10 ChatChunk message for streaming responses
message ChatChunk {
  string token = 1;    #A
  string model = 2;    #B
  optional string finish_reason = 3;    #C
  optional TokenUsage usage = 4;
}

Inspect the message definition showing required token and model fields alongside optional finish_reason and usage fields. section 3.3.2

How Designing AI Systems covers it

How deep it goes
Defines fields and lifecycle semantics, then traces provider adapters through gRPC and SSE, without implementing the adapters.
Examples
Uses OpenAI deltas, Anthropic iterators, vLLM-compatible output, fallback model changes, and a protobuf message definition.
What it assumes
Assumes provider streaming formats and service transport concepts, while standardizing their differences behind one message structure.
What you can do after
Apply it via an SDK streaming generator to append and display model output progressively while tracking final status and usage.
How it connects
Connects provider adapters, Server-Sent Events, gRPC streaming, fallback chains, token usage, and model services.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.