liveBook · Manning

Model service

As explained in Designing AI Systems

A model service is a platform service that provides a unified interface to multiple AI model providers while handling provider-specific APIs, routing, resilience, and operational concerns.

A model service gives applications a unified interface to AI models while handling provider-specific APIs and operational concerns. Instead of embedding provider logic in each workflow, applications request a model through the platform; adapters translate requests and responses. Supported providers can then be changed without rewriting application logic. section 1.4 section 3.2.1

A production contract extends beyond chat. A gRPC interface defines inference and streaming, model discovery, prompt management, custom model registration, and embeddings. The contract specifies operations and data, not how providers, prompt storage, or custom deployments are implemented. section 3.1.5

Listing 3.5 Model service gRPC definition
syntax = "proto3";
 
package ai.platform.models.v1;
 
service ModelService {
  // Core inference
  rpc Chat(ChatRequest) returns (ChatResponse);    #A
  rpc ChatStream(ChatRequest) 
➥returns (stream ChatChunk);    #A
  
  // Discovery
  rpc ListModels(ListModelsRequest) 
➥returns (ListModelsResponse);    #B
  rpc GetModelCapabilities(GetCapabilitiesRequest) 
➥returns (ModelCapabilities);    #B
  
  // Prompt management
  rpc RegisterPrompt(RegisterPromptRequest) returns 
➥(RegisterPromptResponse);
  rpc GetPrompt(GetPromptRequest) returns (Prompt);
  rpc ListPrompts(ListPromptsRequest) returns (ListPromptsResponse);
  
  // Custom model registry
  rpc RegisterModel(RegisterModelRequest) 
➥returns (RegisterModelResponse);    #C
  rpc ListRegisteredModels(ListRegisteredModelsRequest) returns 
➥(ListRegisteredModelsResponse);
  rpc GetModelStatus(GetModelStatusRequest) returns (ModelStatus);
  
  // Embeddings
  rpc Embed(EmbedRequest) returns (EmbedResponse);    #D
  rpc ListEmbeddingModels(ListEmbeddingModelsRequest)
      returns (ListEmbeddingModelsResponse);    #D
}

The service definition shows how the five concerns become callable gRPC operations. section 3.1.5

Provider adapters normalize differences in message structure, system prompts, response parsing, errors, and token usage. The platform uses an OpenAI-style message format as its standard; adapters translate where providers differ. Usage normalization matters because cost attribution must remain comparable across providers. section 3.10 section 3.2.1

Table 3.1 Variations in LLM provider formats
Aspect
OpenAI
Anthropic
Google (Gemini)
System prompt
In messages array as role: "system"
Separate system parameter
In messages array
Message structure
{role, content}
{role, content}
{role, parts}
Response location
choices[0].message.content
content blocks array
candidates[0].content
Token field names
prompt_tokens, completion_tokens
input_tokens, output_tokens
promptTokenCount, candidatesTokenCount

Compare the provider-specific fields an adapter must translate. section 3.2.1

Production behavior belongs behind the interface:

  • Route requests according to cost, load, and capability requirements; support structured outputs and multimodal inputs where needed. section 3.0 section 3.10
  • Let applications configure retries and fallback policies while the service executes them. Different workloads can favor availability or fail-fast behavior. section 3.4
  • Enforce platform-wide request and token limits before calls reach providers. User, workflow, and global scopes protect against different forms of excess usage. section 3.6
  • Record actual provider and model, token usage, latency, failures, and workflow or user identifiers for cost attribution and diagnosis. section 3.8.1
  • Normalize streaming chunks in adapters, forward them over gRPC, and let the gateway deliver them to clients as server-sent events. section 3.3.1

Patient intake assistant

A workflow sends a system prompt, conversation history, and a patient question to the model service. The response includes generated content, the model name, and token counts. A self-hosted clinical model can be registered with its endpoint, capabilities, health check, and credential reference, then requested by name through the same interface. section 3.1.1 section 3.1.4

The model service also participates in a wider request flow. A workflow can supply conversation history and retrieved documents for generation, validate the output through guardrails, and send operational measurements to observability. Keeping these responsibilities coordinated avoids treating a model call as the whole application. section 1.5 section 3.8

How Designing AI Systems covers it

How deep it goes
Defines a gRPC contract and develops provider adapters, streaming, resilience, rate limits, and observability.
Examples
Uses a patient intake assistant and a registered self-hosted clinical model to illustrate chat requests and model registration.
What you can do after
Build it from scratch using a service contract, provider adapters, and configurable operational mechanisms.
How it connects
Connects provider abstraction to workflow orchestration, conversation history, semantic search, guardrails, and observability.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.