A model service gives applications a unified interface to AI models while handling provider-specific APIs and operational concerns. Instead of embedding provider logic in each workflow, applications request a model through the platform; adapters translate requests and responses. Supported providers can then be changed without rewriting application logic. section 1.4 section 3.2.1
A production contract extends beyond chat. A gRPC interface defines inference and streaming, model discovery, prompt management, custom model registration, and embeddings. The contract specifies operations and data, not how providers, prompt storage, or custom deployments are implemented. section 3.1.5
Listing 3.5 Model service gRPC definition
syntax = "proto3";
package ai.platform.models.v1;
service ModelService {
// Core inference
rpc Chat(ChatRequest) returns (ChatResponse); #A
rpc ChatStream(ChatRequest)
➥returns (stream ChatChunk); #A
// Discovery
rpc ListModels(ListModelsRequest)
➥returns (ListModelsResponse); #B
rpc GetModelCapabilities(GetCapabilitiesRequest)
➥returns (ModelCapabilities); #B
// Prompt management
rpc RegisterPrompt(RegisterPromptRequest) returns
➥(RegisterPromptResponse);
rpc GetPrompt(GetPromptRequest) returns (Prompt);
rpc ListPrompts(ListPromptsRequest) returns (ListPromptsResponse);
// Custom model registry
rpc RegisterModel(RegisterModelRequest)
➥returns (RegisterModelResponse); #C
rpc ListRegisteredModels(ListRegisteredModelsRequest) returns
➥(ListRegisteredModelsResponse);
rpc GetModelStatus(GetModelStatusRequest) returns (ModelStatus);
// Embeddings
rpc Embed(EmbedRequest) returns (EmbedResponse); #D
rpc ListEmbeddingModels(ListEmbeddingModelsRequest)
returns (ListEmbeddingModelsResponse); #D
}The service definition shows how the five concerns become callable gRPC operations. section 3.1.5
Provider adapters normalize differences in message structure, system prompts, response parsing, errors, and token usage. The platform uses an OpenAI-style message format as its standard; adapters translate where providers differ. Usage normalization matters because cost attribution must remain comparable across providers. section 3.10 section 3.2.1
Table 3.1 Variations in LLM provider formats
|
Aspect
|
OpenAI
|
Anthropic
|
Google
(Gemini)
|
|
System
prompt
|
In messages
array as role: "system"
|
Separate system
parameter
|
In messages
array
|
|
Message
structure
|
{role, content}
|
{role, content}
|
{role, parts}
|
|
Response
location
|
choices[0].message.content
|
content
blocks array
|
candidates[0].content
|
|
Token field
names
|
prompt_tokens, completion_tokens
|
input_tokens, output_tokens
|
promptTokenCount, candidatesTokenCount
|
Compare the provider-specific fields an adapter must translate. section 3.2.1
Production behavior belongs behind the interface:
- Route requests according to cost, load, and capability requirements; support structured outputs and multimodal inputs where needed. section 3.0 section 3.10
- Let applications configure retries and fallback policies while the service executes them. Different workloads can favor availability or fail-fast behavior. section 3.4
- Enforce platform-wide request and token limits before calls reach providers. User, workflow, and global scopes protect against different forms of excess usage. section 3.6
- Record actual provider and model, token usage, latency, failures, and workflow or user identifiers for cost attribution and diagnosis. section 3.8.1
- Normalize streaming chunks in adapters, forward them over gRPC, and let the gateway deliver them to clients as server-sent events. section 3.3.1
Patient intake assistant
A workflow sends a system prompt, conversation history, and a patient question to the model service. The response includes generated content, the model name, and token counts. A self-hosted clinical model can be registered with its endpoint, capabilities, health check, and credential reference, then requested by name through the same interface. section 3.1.1 section 3.1.4
The model service also participates in a wider request flow. A workflow can supply conversation history and retrieved documents for generation, validate the output through guardrails, and send operational measurements to observability. Keeping these responsibilities coordinated avoids treating a model call as the whole application. section 1.5 section 3.8
