liveBook · Manning

Chapter 3: The Model Service: Your platform’s gateway to AI models

What you will learn in chapter 3 of Designing AI Systems

This introduction positions the Model Service as the platform component that turns application requests into AI responses while hiding provider and operational complexity.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction positions the Model Service as the platform component that turns application requests into AI responses while hiding provider and operational complexity.

Key terms: Model Service, Session Service, Data Service, Tool Service, provider, platform.models.chat(), provider adapters, unified interface, OpenAI, Anthropic

3.1 The model service contract

This section introduces the Model Service as the platform gateway to AI models and identifies the contract capabilities required as an AI application moves from prototype to production.

Key terms: Model Service, workflows, AI models, gRPC, protocol buffer contracts, Model Service contract, SDK, API Gateway, generating responses, discovering available models

3.1.1 Generating responses

This section defines the Model Service’s fundamental chat operation: submit a role-based conversation to a chosen model and receive generated text plus usage metadata.

Key terms: Model Service, conversation, messages, system, user, assistant, provider, OpenAI API, credentials, endpoint

3.1.2 Discovering available models

This section shows how model discovery lets workflows query the Model Service for available models and capabilities instead of relying on stale hardcoded model names.

Key terms: model discovery operation, Model Service, available models, capabilities, source of truth, context window, 100,000 tokens, vision support, filter, runtime

3.1.3 Managing system prompts

This section shows how the Model Service turns system prompts from workflow-embedded text into centrally managed, versioned assets.

Key terms: system prompt, prompt registry, prompt-management operations, metadata, managed assets, version, rollback, prompt name, Experimentation Service, comparison

3.1.4 Registering custom models

This section explains how registering a self-hosted custom model lets it participate in the Model Service like a commercial model while retaining platform-wide operations and observability.

Key terms: self-hosted models, Llama, vLLM, data privacy, cost predictability, fine-tuning, model registration, custom model, Model Service, credential

3.1.5 The gRPC contract

This section formalizes the Model Service’s gRPC contract, grouping its operations into five concerns while separating the stable interface from changing implementation details.

Key terms: Model Service, gRPC contract, Chat, ChatStream, ListModels, GetModelCapabilities, embedding, vectors, Embed, ListEmbeddingModels

3.1.6 Request and response structures

This section defines the chat request and response data structures that precisely carry model inputs, advanced controls, generated outputs, and usage information through the Model Service.

Key terms: ChatRequest, model, messages, Message, config, system_prompt_name, tools, ToolDefinition, FunctionDef, JSON Schema

3.2 Provider abstraction

This section introduces provider abstraction as the Model Service machinery that hides differences among AI providers behind one consistent interface.

Key terms: Model Service, provider abstraction, multi-provider support, consistent interface, AI providers

3.2.1 How providers differ

This section shows how incompatible LLM provider APIs create lock-in and why the Model Service must hide those differences behind a unified abstraction.

Key terms: message formats, response structures, parameter names, API patterns, system prompts, token counting, system prompt, messages array, role: system, separate parameter

3.2.2 The unified provider interface

This section defines a unified provider interface and adapter architecture that lets applications use different AI model providers through the Model Service without provider-specific code.

Key terms: adapter pattern, unified interface, provider adapters, Model Service, routing, fallback logic, configuration, ModelProvider, synchronous chat, streaming chat

3.2.3 OpenAI message format as platform standard

This section establishes the OpenAI message format as the Model Service’s canonical internal representation because it is a familiar de facto standard across providers and tools.

Key terms: canonical representation, OpenAI message format, role, content, Model Service, de facto industry standard, OpenAI-compatible endpoint, vLLM, translation, provider

3.2.4 What adapters do

This section explains how provider adapters hide AI-provider differences by translating requests, responses, errors, and structured-output requirements into uniform platform behavior.

Key terms: adapter, message translation, parameter mapping, response normalization, error translation, ModelProviderError

3.2.5 The adapter implementation pattern

This section shows how provider adapters translate between the platform’s unified interface and each model provider’s API so hosted and self-hosted models can be used interchangeably.

Key terms: adapter, unified interface, provider format, translation logic, system message extraction, content blocks, choices[0].message.content, parts format, candidates[0].content, self-hosted models

3.3 Streaming responses

Streaming delivers model output incrementally to improve perceived responsiveness and allow workflows to stop unsafe or unwanted generation early.

Key terms: streaming, time-to-first-token, time-to-complete, early termination, abort

3.3.1 The streaming architecture

This section traces how streamed model tokens move from provider APIs through adapters, the Model Service, and the API Gateway to an SSE-connected client.

Key terms: provider adapter, Model Service, gRPC streaming, API Gateway, Server-Sent Events, SSE, platform representation, delta.content, text_stream, OpenAI-compatible

3.3.2 The ChatChunk message

This section defines the provider-independent ChatChunk message that gives streaming consumers a consistent way to append text, detect completion, identify models, and account for usage.

Key terms: ChatChunk, adapter layer, streaming responses, token, model, fallback chain, tokenization, finish_reason, usage, stop

3.3.3 Streaming and error handling

This section explains how the Model Service preserves partial streamed output when failures occur and why mid-stream failures restrict fallback recovery.

Key terms: final chunk, finish_reason, error, mid-stream error, partial response, error indication, mid-stream errors, fallback behavior, synchronous requests, complete failures

3.4 Resilience: fallbacks and retries

This section introduces application-defined retry and fallback policies that the Model Service executes centrally to handle provider failures without workflow-level error handling.

Key terms: resilience primitives, retries, fallbacks, error handling, provider outages, rate limits, retry configuration, fallback configuration, Model Service, workflow

3.4.1 Retry configuration

This section explains how the Model Service configures retries for transient provider failures while avoiding wasteful retries for permanent errors.

Key terms: transient failures, permanent failures, retry_on, rate limit errors, timeout errors, service unavailable errors, RetryConfig, max_retries, initial_delay, exponential_backoff

3.4.2 Fallback configuration

This section explains how fallback configuration moves failed model requests through an ordered provider chain while handling non-recoverable errors and recording the provider that ultimately responds.

Key terms: FallbackConfig, providers, retry_config, fallback providers, fail_on, invalid request errors, content policy violations, provider field, logging, cost attribution

3.4.3 Configuration examples

This section shows how resilience configuration should vary for patient-facing, analytics, and cost-sensitive applications.

Key terms: aggressive retries, fallback chain, Claude, local Llama model, on-premises, exponential backoff, fallbacks disabled, rate limits, pipeline level, timeouts

3.5 Routing strategies

This section introduces routing as the Model Service’s pre-request provider-selection responsibility and explains why common routing policies belong centrally in the platform.

Key terms: routing, fallbacks, provider selection, Model Service, routing strategies, cost-aware routing, load balancing, feature-based selection, custom routing logic, provider

3.5.1 Routing configuration

This section explains how application-defined routing configuration lets the Model Service choose a provider when a workflow does not explicitly name one.

Key terms: RoutingConfig, strategy, default_provider, provider selection, application-side, workflows, Model Service, explicit provider, routing bypassed, spending state

3.5.2 Cost-aware routing

This section explains how the Model Service routes requests among models based on budget status and task complexity to control AI spending.

Key terms: cost-aware routing, decision tree, budget limit, local models, API models, task complexity, token counts, provider pricing, running totals, budget resets

3.5.3 Load-based routing

This section explains how the Model Service routes requests to the least-loaded provider to increase throughput and avoid provider rate limits, while noting when output consistency makes such routing unsuitable.

Key terms: load-based routing, Model Service, active requests, least-loaded provider, counters, increment, decrement, rate limits, throughput, fallbacks

3.5.4 Feature-based routing

This section explains how feature-based routing selects only model providers that satisfy every capability required by a request and rejects unsupported requests early.

Key terms: feature-based routing, capabilities, providers, vision, function calling, context windows, capability matrix, set intersection, required features, capable providers

3.5.5 Combining patterns

This section explains how the Model Service combines feature, cost, and load-routing strategies into an ordered decision process.

Key terms: combined routing strategy, feature requirements, hard constraints, cost logic, soft preferences, load balancing, cost preferences, vision-capable providers, platform.models.chat(messages), Model Service

3.6 Rate limiting

This section explains how platform-level rate limiting protects an organization from runaway model usage and cost while distinguishing policy enforcement from provider throttling.

Key terms: Model Service, platform-level rate limiting, provider rate limits, fallback chains, scope, per-user limits, per-workflow limits, global limits, requests per minute, tokens per minute

3.7 Caching for cost and performance

Introduces caching as a cost- and latency-reduction technique for LLM applications that requires choosing among multiple strategies and trade-offs.

Key terms: caching, LLM applications, caching strategies, trade-offs, API costs, latency

3.7.1 Two levels of caching

This section distinguishes application-level response caching from provider-level prompt caching and explains how exact and semantic response caching can work together to reduce latency and cost.

Key terms: response caching, exact-match caching, semantic caching, embedding, similarity threshold, cache hit, identical requests, zero cost, sub-millisecond latency, varied responses

3.7.2 The cache interface

This subsection defines configurable response and prompt caching in the Model Service, including cache-key correctness, eviction behavior, and provider-specific prompt-cache handling.

Key terms: response caching, cache keys, hash, model, messages, configuration, per request, LRU eviction, capacity, TTL expiration

3.7.3 Monitoring cache effectiveness

This section explains how the Model Service measures whether response and prompt caches save money and uses those measurements to tune caching and attribute costs accurately.

Key terms: response cache hit rate, TTL, temperature, cache configuration, prompt cache savings, response metadata, response cache hit, prompt cache hit, full cache miss, discounted rate

3.8 Observability: Cost tracking and metrics

This section introduces how the Model Service records per-request metrics and sends them to the Observability Service so AI spending can be traced and optimized.

Key terms: Model Service, comprehensive metrics, Observability Service, token consumption, workflows

3.8.1 What the model service tracks

This section defines the per-request metrics the Model Service records to measure model usage, actual execution, performance, failures, and attribution.

Key terms: prompt tokens, completion tokens, total tokens, response cache, prompt caching, provider, model, fallbacks, cost attribution, time to first token

3.8.2 Feeding the observability service

The Model Service publishes per-request metrics to the Observability Service so teams can attribute AI costs, analyze usage, and detect operational problems in real time.

Key terms: Model Service, Observability Service, metrics, per-request metrics, provider, model, workflow, user, time, cost-effectiveness

3.8.3 Enabling informed decisions

This section shows how Model Service observability data enables teams to attribute costs, diagnose problems, detect anomalies, and make evidence-based optimization decisions.

Key terms: token usage by workflow, token usage by model, cost per user, targeted optimization, GPT-4o-mini, GPT-4, average response time, fallbacks, rate-limiting, primary provider

3.9 Integrating with the SDK

This section positions the SDK's ModelClient as the layer that turns developers' simple model calls into gRPC requests routed through the API Gateway to the Model Service.

Key terms: SDK, ModelClient, Protocol Buffer messages, gRPC, Model Service, platform.models.chat(messages), GenAIPlatform, API Gateway, gRPC channel, client object

3.9.1 The ModelClient

This subsection explains how the SDK creates a ModelClient that routes model requests through the Gateway and wraps generated gRPC stubs in a Python-friendly interface.

Key terms: ModelClient, Protocol Buffer requests, Protocol Buffer responses, Python objects, service_name="models", routing, API Gateway, model service, _stub, ModelServiceStub

3.9.2 Method implementation pattern

This section traces how ModelClient.chat translates a Python call into a routed gRPC request and returns a Python-friendly response.

Key terms: ModelClient, Protocol Buffer, ChatMessage, ChatResponse, _dict_to_proto_msg, _stub.Chat(), model name, routing logic, cost-aware routing, load-based routing

3.9.3 Streaming support

This section explains how the SDK exposes Model Service streaming as a Python generator that progressively yields response chunks to workflow code.

Key terms: platform.models.chat_stream, generator, chunks, gRPC stream, ModelClient, Model Service, Gateway, provider

3.9.4 The complete picture

This section traces a ModelClient chat request from Python workflow code through the platform and provider, showing how the platform hides distributed integration complexity.

Key terms: GenAIPlatform, ModelClient, ChatRequest, Protocol Buffer, gRPC, Gateway, requested model, OpenAI adapter, system_prompt_name, registered prompt

3.10 Summary

This summary consolidates how the Model Service provides a provider-independent model gateway with streaming, resilience, routing, cost controls, observability, and SDK access.

Where this is used later

Concepts explained in this chapter

← Chapter 2: Building with the platform: SDK and API design Chapter 4: The Session Service: Teaching your AI to remember →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.