ChatChunk is the platform’s normalized unit for streamed model output. It carries one text fragment, the producing model, an optional completion reason, and optional token usage so downstream services can process streams without provider-specific logic. section 3.3.2
Use the fields to assemble output, attribute generation, and finalize operations:
- token contains the arriving text fragment. It may be a word, part of a word, or punctuation; consumers generally append it as received. section 3.3.2
- model identifies the producer for each fragment, allowing consumers to detect mid-stream fallback changes and attribute cost. section 3.3.2
- finish_reason appears on the final chunk and records stop, length, tool_calls, content_filter, or error termination. section 3.3.2
- usage typically appears only at completion because token counts become available after generation ends, making streaming cost tracking retrospective. section 3.3.2
Provider-normalized streaming
A provider adapter converts OpenAI delta objects, Anthropic text-stream events, or compatible self-hosted output into ChatChunk messages. The Model Service forwards them through gRPC streaming, while the API Gateway emits Server-Sent Events. A client appends each token until a final chunk supplies finish_reason and, typically, usage. section 3.3.1 section 3.3.2
Listing 3.10 ChatChunk message for streaming responses
message ChatChunk {
string token = 1; #A
string model = 2; #B
optional string finish_reason = 3; #C
optional TokenUsage usage = 4;
}Inspect the message definition showing required token and model fields alongside optional finish_reason and usage fields. section 3.3.2
