Chapter 3: The Model Service: Your platform’s gateway to AI models
This introduction positions the Model Service as the platform component that turns application requests into AI responses while hiding provider and operational complexity.
After this chapter you will be able to
- Model Service role. Explain why the Model Service is the central response-producing component of the platform, coordinating context, provider selection, and transformation of a user message into a reply.
- Unified model access. Describe how a single platform interface lets application code request models such as GPT-4, Claude, or a local Llama model without changing code for provider-specific differences.
- Application-platform responsibility boundary. Distinguish application-side intent from platform-side execution: applications choose requested models, messages, temperature, token limits, and sometimes routing configuration, while the Model Service translates formats, executes routes and fallbacks, caches, and tracks costs.
- Model Service platform role. Explain that the Model Service is the platform component through which workflows interact with AI models for tasks such as response generation, document analysis, and text classification. Describe why its central, heavily used position makes its interface important to the rest of the platform.
- gRPC contract as specification. Describe the Model Service contract as a gRPC protocol buffer interface that specifies available operations, request data, and expected responses. Explain that this contract is the promise used by SDK methods, the API Gateway, and other platform services to integrate with the Model Service.
- Production model capabilities. State the four capabilities the Model Service must expose for a production-oriented AI application: generating responses, discovering available models, managing system prompts, and registering custom models. Explain the practical need behind each capability: adaptable multi-provider model choice, centrally maintainable prompts, and custom models that work like commercial ones.
- Chat request and response contract. Describe the basic Model Service operation: a client specifies a model and sends a conversation as a list of messages, then receives the model’s generated response. Explain the roles of system, user, and assistant messages in that conversation.
- Provider complexity concealment. Explain that the simple generation interface hides provider-specific work: obtaining credentials and endpoint information, translating the platform request into the provider’s format, making the network call, translating the result back, and handling errors.
- Capability-based model discovery. Explain how the model discovery operation returns available models together with their capabilities, making the platform the source of truth for what models can be used.
- Runtime capability filtering. Describe how a workflow can filter discovered models by required capabilities at runtime, such as a context window above 100,000 tokens or vision support, and then select a suitable model.
- Prompt registry management. Explain why the Model Service stores system prompts in a prompt registry rather than having workflows embed prompt text directly. Describe that prompts are registered by name with metadata and then shared across workflows as managed assets.
- Prompt versioning and rollback. Describe how registering new prompt versions preserves older versions for comparison or rollback. Explain that named, versioned prompts allow tests to compare prompt behavior without creating or changing separate workflow implementations.
- Custom model registration. Describe how registering a custom model tells the Model Service its endpoint, credential, capabilities, and health check so the platform can use it as a known model.
- First-class custom model operation. Explain that a registered custom model is invoked through the same Model Service interface and response format as commercial models, preserving logging, usage tracking, health visibility, fallbacks, and configuration-based A/B testing.
- Five contract operation categories. Describe the five categories of Model Service operations and connect each category to the methods in the gRPC contract: response generation, model discovery, prompt management, custom-model registration, and embeddings.
- Embedding service responsibility. Explain what an embedding represents, why semantically similar text has nearby embeddings, and why embedding generation belongs in the Model Service rather than a separate service.
- Contract and implementation separation. Explain that the gRPC contract defines what the Model Service offers and how clients call it, not how providers, prompt storage, custom-model execution, or health checks are implemented; justify why a stable contract lets implementations change without breaking workflows.
- Chat request composition. Describe the fields of a ChatRequest and what information each supplies to the Model Service, including model selection, conversation messages, generation configuration, and named system prompts.
- Advanced request controls. Explain how tool definitions and response_format extend a chat request, including why structured JSON output can be used directly by downstream systems.
- Chat response contents. Describe what a ChatResponse reports: generated content, the actual model and provider used, token usage, optional tool calls, and the reason generation finished.
- Multi-provider abstraction. Explain why the Model Service abstracts AI providers behind a consistent interface, despite providers performing similar model tasks in different ways.
- Provider API variations. Describe how major LLM providers differ in message formats, system-prompt handling, response structures, token-usage fields, and API conventions. Explain that these independently evolved differences obstruct a unified application interface.
- Provider lock-in costs. Explain why directly integrating application code with one provider creates vendor lock-in: adding a provider or fallback requires changing message construction, API calls, response parsing, and error handling. Describe how the Model Service abstraction replaces application rewrites with an adapter plus configuration changes, enabling fallback, provider experimentation, A/B tests, and data-driven model choices.
- Provider adapter abstraction. Explain how the adapter pattern gives applications one unified interface while provider-specific adapters translate that interface to each provider's API. State that adding a provider requires a new adapter rather than changes to application code.
- Three-layer request architecture. Describe the three layers of the Model Service architecture: applications request a named model from the Model Service, the service routes according to configuration and applies user-defined fallbacks, and provider adapters handle provider API specifics.
- Canonical message format. Explain that the platform represents a conversation as an array of message objects, each carrying a role and content, using the OpenAI message format as its canonical internal format.
- De facto standard rationale. Justify the choice of OpenAI’s format by explaining that it is a de facto industry standard, not a preference for OpenAI, and that this reduces translation for OpenAI and OpenAI-compatible vLLM while giving developers a familiar structure.
- Adapter translation responsibilities. Explain the five responsibilities of a provider adapter: message translation, parameter mapping, response normalization, error translation, and structured output translation. Describe how these responsibilities let the platform present one consistent model-service interface despite provider-specific APIs.
- Adapter translation flow. Describe the common provider-adapter flow: accept unified platform input, translate it into a provider-specific request, call that provider’s API, and translate the provider response back into the unified format. Explain that the flow stays the same across providers even though each provider’s message and response structures differ.
- Self-hosted model equivalence. Explain why self-hosted models can use the same abstraction: vLLM, Text Generation Inference, and Ollama expose OpenAI-compatible APIs, so their adapters resemble the OpenAI adapter, with endpoint configuration and authentication as the main differences. State that this enables switching between cloud and local models without application code changes.
- Streaming user experience. Explain how streaming changes a model interaction from waiting for a completed response to seeing the response unfold as it is generated, and why early output improves perceived speed.
- Early generation termination. Describe how streaming enables a workflow to inspect output as it arrives and abort generation early when it detects problematic content, avoiding the generation of tokens that will be rejected.
- End-to-end streaming pipeline. Trace the streaming path from a model provider to a frontend client: provider tokens are normalized by an adapter, streamed by the Model Service over gRPC to the API Gateway, and delivered by the gateway as Server-Sent Events.
- Provider-independent streaming chunks. Explain how ChatChunk gives streaming consumers one common message format even though providers emit tokens in different native formats. Describe how the adapter layer normalizes provider-specific streaming events so workflow code does not need provider-specific conditional logic.
- Chunk content and model identity. Describe the token and model fields in a ChatChunk. Explain that token text can be a word, part of a word, or punctuation and is normally appended as received, while the model field identifies the producer of each chunk, including chunks that may come from different models after a fallback.
- Stream completion and usage signals. Explain the final-chunk semantics of finish_reason and usage. State that finish_reason marks stream completion and can report stop, length, tool_calls, content_filter, or error, while final usage contains token counts only after generation completes, so streaming cost tracking occurs after the fact.
- Mid-stream error signaling. Describe how the Model Service reports a provider failure after streaming has started: it sends a final chunk whose finish_reason is set to error rather than attempting to return a normal error response.
- Partial response preservation. Explain that output received before a mid-stream failure remains valid up to the error point, so clients can show the partial response together with an error indication rather than losing everything.
- Resilience policy choices. Explain why model-provider outages, rate limits, and network timeouts require resilience policies, and describe how the appropriate retry or fallback behavior depends on the application’s availability and failure-tolerance needs.
- Platform-side policy execution. Describe the division of responsibility in resilience handling: the application configures retry, fallback, and error-handling policy, while the Model Service executes that policy without requiring the workflow to catch errors or implement retry loops.
- Retriable error classification. Distinguish transient errors that should be retried from permanent errors that should not. State that rate limits (HTTP 429), timeouts, and service-unavailable errors (HTTP 503) are retry candidates, whereas invalid requests (HTTP 400) are not.
- Retry timing configuration. Describe how max_retries, initial_delay, exponential_backoff, and max_delay control retry attempts and waiting time. Explain that exponential backoff lengthens successive waits, such as one, two, then four seconds, while max_delay caps that growth.
- Ordered provider fallback chain. Describe how the providers field defines fallback providers in preference order after the primary provider fails, and explain that each provider receives its configured retry attempts before the service advances to the next provider.
- Immediate-failure error policy. Explain how fail_on identifies errors that should stop the request immediately rather than trigger fallbacks, such as malformed requests and potentially content policy violations.
- Availability-focused resilience strategy. Describe the patient-facing configuration: aggressive short retries and a deep fallback chain prioritize continued responses over provider failures. Explain that cloud alternatives and a local Llama last resort preserve availability during cloud outages and can keep sensitive patient data on-premises.
- Analytics pipeline failure handling. Explain why an analytics batch pipeline uses longer exponential-backoff retries with fallbacks disabled. Describe how surfaced failures let the pipeline decide whether to skip an item, queue it for later, or alert an operator.
- Routing versus fallbacks. Distinguish routing from fallbacks by explaining that routing selects a provider before a request runs, whereas fallbacks handle provider failures after they occur.
- Configurable routing strategies. Describe the routing choices the Model Service can make when a workflow does not name a provider, including a fixed provider, cost-aware selection, load balancing, and capability-based selection. Explain that workflows can use built-in strategies, combine them, or supply custom routing logic.
- Centralized routing state. Explain why routing belongs in the Model Service: provider selection depends on cross-request state such as spending against a budget, active request counts, and provider capability matrices, which the service maintains and exposes to built-in or custom routing logic.
- Routing configuration roles. Describe how RoutingConfig controls provider selection, including the strategy field, default_provider fallback, and per-strategy configuration fields.
- Routing ownership and bypass. Explain when provider routing runs and distinguish workflow-side routing preferences from platform-side routing execution and state management.
- Cost-aware routing decisions. Explain how cost-aware routing selects a provider using both current budget spending and the complexity of the task. Describe the decision pattern: use the cheapest option near the budget limit, inexpensive API models for routine work, and powerful expensive models only for complex work that justifies their cost.
- Least-loaded provider routing. Explain how load-based routing selects a provider for each incoming request. Describe how the Model Service tracks active requests per provider and sends each new request to the provider with the lowest current load.
- Applicability and consistency tradeoff. Describe when load-based routing is useful and when it may be inappropriate. Explain that it helps use total capacity and avoid rate-limit-triggered fallbacks when providers have rate limits and similar latency, but different model families can produce inconsistent outputs; for consistency-sensitive work, balance across API keys for the same provider instead.
- Capability-based provider selection. Explain how feature-based routing chooses providers according to the capabilities a particular request requires, such as vision, function calling, large context windows, or specific output formats.
- Capability matrix intersection. Describe how the model service uses a capability matrix and set intersection to find providers that support all required features, with each requirement narrowing the eligible set.
- Combined routing sequence. Describe how combined routing evaluates feature requirements first, cost logic second, and load balancing last when selecting a model provider. Explain that each stage narrows the remaining provider options.
- Platform-level rate limiting. Explain why the Model Service enforces rate limits before requests reach providers, rather than leaving limiting to providers or individual applications. Describe how a platform-wide view protects against cross-application abuse and runaway spending.
- Limit scopes and consumption dimensions. Describe the per-user, per-workflow, and global scopes for rate limits, and distinguish request-per-minute limits from token-per-minute limits. Explain why both request and token limits may be needed to prevent automated abuse and expensive oversized contexts.
- Rate-limit error handling. Distinguish platform rate-limit responses from provider-originated rate limits in the retry and fallback flow. Describe the Retry-After indicator and explain why platform limits must not trigger a fallback to another provider.
- Multi-level caching trade-offs. Explain that caching can reduce both API costs and response latency in LLM applications, but is not a single technique. Describe that multiple caching strategies operate at different levels and must be chosen according to their distinct trade-offs.
- Response caching strategies. Describe how response caching stores and returns complete model responses, and distinguish exact-match caching from semantic caching. Explain why an application can check an exact match first and then use semantic similarity after an exact-match miss.
- Provider prompt caching. Distinguish provider-level prompt caching from response caching. Describe how providers reuse previously seen prompt content to discount cached tokens while the application still makes the API call, and explain how prompt caching complements response caching after a response-cache miss.
- Response cache key composition. Explain how the Model Service generates response-cache keys from the model, messages, and configuration, and why a changed temperature must produce a different key even when the messages are identical.
- Provider-level prompt caching. Distinguish provider-level prompt caching from response caching, and describe how the adapter layer handles provider-specific controls while applications choose per request whether to enable that caching.
- Response cache hit-rate monitoring. Explain what response cache hit rate measures and how it guides response-cache configuration. Describe how a much lower-than-expected hit rate can indicate variable queries, an overly short TTL, or excessive temperature.
- Cache-aware cost attribution. Describe how prompt-cache savings are obtained from provider response metadata and aggregated across requests. Distinguish the cost of a response-cache hit, a prompt-cache hit, and a full cache miss, and explain why the service must record the path each request took.
- Per-request cost observability. Explain that the Model Service tracks comprehensive metrics for every model request and feeds them to the platform's Observability Service. Describe how this makes provider costs traceable to the workflows, features, or users responsible for token consumption.
- Token usage metrics. Describe the token metrics recorded for each Model Service response, including prompt, completion, total, response-cache, and prompt-cache tokens. Explain how these metrics support cost calculations because providers charge per token and caches can eliminate or reduce that cost.
- Actual provider and model. Explain why metrics record the provider and specific model that actually handled a request rather than only the model originally requested. Relate this to fallback behavior and accurate cost attribution across providers with different rates.
- Latency and outcome tracking. Describe the timing and outcome metrics for a request: time to first token, total duration, time spent retrying or attempting fallbacks, and success or categorized failure. Explain how first-token latency reflects streaming user experience, total duration supports throughput and timeout planning, and error categories reveal recurring versus transient problems.
- Observability metric publication. Describe how the Model Service records metrics locally on every request and publishes them to the platform Observability Service for cross-platform aggregation.
- Multi-dimensional usage analysis. Explain that the Observability Service aggregates Model Service data by provider, model, workflow, user, and time, enabling teams to compare spending, effectiveness, resource consumption, and usage trends.
- Cost attribution and decisions. Explain how observed token usage and model choices are attributed back to workflows and features, so teams can identify the cause of a budget increase and decide whether higher model quality justifies higher cost.
- Cost driver analysis. Explain how token usage by workflow and model, plus cost per user, identify where AI spending comes from and support targeted cost optimization. Describe the example of using a cheaper model for short document summaries while reserving GPT-4 for complex documents.
- Performance bottleneck diagnosis. Describe how response-time metrics reveal latency regressions and how drilling into the data can connect a slowdown to frequent fallbacks caused by primary-provider rate limiting. Explain why load-based routing across multiple providers can address that particular bottleneck.
- Evidence-based model routing. Explain how comparative quality, cost, and error-rate data can guide model and provider selection instead of guesswork. Describe the example where Claude Sonnet can handle classification when it has similar quality to GPT-4 at 60% lower cost, while provider-specific errors can also change routing.
- SDK-to-model-service translation. Explain how the SDK lets a developer call platform.models.chat(messages) without interacting with gRPC directly, translating Python method calls into Protocol Buffer messages for the Model Service.
- ModelClient responsibilities. Describe the three responsibilities of a ModelClient: connecting to the Gateway, constructing Protocol Buffer requests for model operations, and converting Protocol Buffer responses into Python objects usable by application code.
- Generated gRPC stub. Describe the generated ModelServiceStub as code produced when the .proto contract is compiled, providing Python methods for gRPC operations such as Chat, ChatStream, and ListModels while handling serialization and network communication.
- Three-phase client method pattern. Describe the three phases shared by SDK client methods: converting Python inputs to Protocol Buffer messages, making the gRPC call with routing metadata, and converting the Protocol Buffer response back into a Python dataclass. Explain that this hides Protocol Buffer details from workflow code.
- SDK streaming generator. Describe how calling platform.models.chat_stream returns a generator whose iterations progressively yield streamed response chunks as Python objects. Explain that each iteration waits for the next chunk rather than returning one complete response.
- End-to-end chat request flow. Trace a chat request from Sarah's workflow code through the SDK, Gateway, Model Service, OpenAI adapter, and back to a Python response. Include the key request transformations and routing decisions described in the complete flow.
- Intent-focused SDK abstraction. Explain the developer experience produced by this stack: Sarah makes one readable method call and receives a Python object, while the platform hides serialization, gRPC channels, Gateway routing, provider adaptation, and external API calls.
What it builds on
- Your AI platform blueprint: what we're building (chapter 1) · The platform blueprint and its separation into specialized services.
- Building the SDK (chapter 2) · The SDK as the application-facing access layer for platform services.
- Designing the ideal developer experience (chapter 2) · The developer experience contract that makes platform capabilities easy for applications to use.
- One workflow, one service (chapter 2) · The idea that each workflow runs as its own service and needs platform services to perform shared capabilities.
- Tracing the Complete Flow (chapter 2) · The traced end-to-end flow through platform communication components.
- The communication protocol (chapter 2) · The platform communication protocol for exposing and invoking workflow capabilities.
- The internal communication challenge (chapter 2) · The internal communication architecture between platform services.
- Connection to the API Gateway (chapter 2) · The SDK's connection to the API Gateway.
Section by section
Introduction
This introduction positions the Model Service as the platform component that turns application requests into AI responses while hiding provider and operational complexity.
- Model Service role: Explain why the Model Service is the central response-producing component of the platform, coordinating context, provider selection, and transformation of a user message into a reply.
Why it matters: It clarifies where model inference belongs among the platform's specialized services. - Unified model access: Describe how a single platform interface lets application code request models such as GPT-4, Claude, or a local Llama model without changing code for provider-specific differences.
Why it matters: A unified interface lets applications change or use multiple model providers without taking on provider-specific integration work. - Application-platform responsibility boundary: Distinguish application-side intent from platform-side execution: applications choose requested models, messages, temperature, token limits, and sometimes routing configuration, while the Model Service translates formats, executes routes and fallbacks, caches, and tracks costs.
Why it matters: This boundary lets developers express what they need while the platform owns repeated operational complexity.
Key terms: Model Service, Session Service, Data Service, Tool Service, provider, platform.models.chat(), provider adapters, unified interface, OpenAI, Anthropic
3.1 The model service contract
This section introduces the Model Service as the platform gateway to AI models and identifies the contract capabilities required as an AI application moves from prototype to production.
- Model Service platform role: Explain that the Model Service is the platform component through which workflows interact with AI models for tasks such as response generation, document analysis, and text classification. Describe why its central, heavily used position makes its interface important to the rest of the platform.
Why it matters: A single platform gateway keeps model interactions consistent as many workflows and services depend on them. - gRPC contract as specification: Describe the Model Service contract as a gRPC protocol buffer interface that specifies available operations, request data, and expected responses. Explain that this contract is the promise used by SDK methods, the API Gateway, and other platform services to integrate with the Model Service.
Why it matters: A clear service contract lets independently built platform components communicate predictably. - Production model capabilities: State the four capabilities the Model Service must expose for a production-oriented AI application: generating responses, discovering available models, managing system prompts, and registering custom models. Explain the practical need behind each capability: adaptable multi-provider model choice, centrally maintainable prompts, and custom models that work like commercial ones.
Why it matters: These capabilities prevent a prototype's hard-coded model and prompt choices from becoming a production maintenance problem.
Key terms: Model Service, workflows, AI models, gRPC, protocol buffer contracts, Model Service contract, SDK, API Gateway, generating responses, discovering available models
3.1.1 Generating responses
This section defines the Model Service’s fundamental chat operation: submit a role-based conversation to a chosen model and receive generated text plus usage metadata.
- Chat request and response contract: Describe the basic Model Service operation: a client specifies a model and sends a conversation as a list of messages, then receives the model’s generated response. Explain the roles of system, user, and assistant messages in that conversation.
Why it matters: This contract is the common foundation for every application that asks a model to generate a response. - Provider complexity concealment: Explain that the simple generation interface hides provider-specific work: obtaining credentials and endpoint information, translating the platform request into the provider’s format, making the network call, translating the result back, and handling errors.
Why it matters: A stable platform interface lets workflows use models without each workflow implementing provider integration details. - Source-level usage metadata: Explain that a response includes the model name and token counts as well as generated text, and justify capturing token counts at the source for cost attribution, usage analysis, and reliable observability.
Why it matters: Capturing actual usage with each response avoids unreliable cost estimates and supports organization-wide cost tracking.
Key terms: Model Service, conversation, messages, system, user, assistant, provider, OpenAI API, credentials, endpoint
3.1.2 Discovering available models
This section shows how model discovery lets workflows query the Model Service for available models and capabilities instead of relying on stale hardcoded model names.
- Capability-based model discovery: Explain how the model discovery operation returns available models together with their capabilities, making the platform the source of truth for what models can be used.
Why it matters: Discovery lets applications adapt when the organization adds, removes, or changes available models. - Runtime capability filtering: Describe how a workflow can filter discovered models by required capabilities at runtime, such as a context window above 100,000 tokens or vision support, and then select a suitable model.
Why it matters: Capability filtering allows one workflow to select models appropriate to different tasks without breaking when model offerings change. - Model-list caching strategy: State that the SDK client should cache the model list when it initializes and refresh it periodically, such as every few minutes, rather than querying before every request.
Why it matters: Caching avoids adding a network round-trip to hot request paths while still allowing the client to learn about occasional availability changes.
Key terms: model discovery operation, Model Service, available models, capabilities, source of truth, context window, 100,000 tokens, vision support, filter, runtime
3.1.3 Managing system prompts
This section shows how the Model Service turns system prompts from workflow-embedded text into centrally managed, versioned assets.
- Prompt registry management: Explain why the Model Service stores system prompts in a prompt registry rather than having workflows embed prompt text directly. Describe that prompts are registered by name with metadata and then shared across workflows as managed assets.
Why it matters: A central prompt registry prevents inconsistent copies of critical instructions and makes prompt changes manageable. - Prompt versioning and rollback: Describe how registering new prompt versions preserves older versions for comparison or rollback. Explain that named, versioned prompts allow tests to compare prompt behavior without creating or changing separate workflow implementations.
Why it matters: Versioned prompts let teams improve behavior safely while retaining a known working version.
Key terms: system prompt, prompt registry, prompt-management operations, metadata, managed assets, version, rollback, prompt name, Experimentation Service, comparison
3.1.4 Registering custom models
This section explains how registering a self-hosted custom model lets it participate in the Model Service like a commercial model while retaining platform-wide operations and observability.
- Self-hosted model tradeoffs: Explain the privacy, cost-predictability, and customization advantages of self-hosted models, along with the operational complexity and possible capability gaps they introduce.
Why it matters: Knowing these tradeoffs helps teams decide when owning model infrastructure is worth it. - Custom model registration: Describe how registering a custom model tells the Model Service its endpoint, credential, capabilities, and health check so the platform can use it as a known model.
Why it matters: Registration avoids making each workflow handle a custom model as a separate integration. - First-class custom model operation: Explain that a registered custom model is invoked through the same Model Service interface and response format as commercial models, preserving logging, usage tracking, health visibility, fallbacks, and configuration-based A/B testing.
Why it matters: A uniform model interface keeps application code portable while operations teams manage every model through the same platform tooling.
Key terms: self-hosted models, Llama, vLLM, data privacy, cost predictability, fine-tuning, model registration, custom model, Model Service, credential
3.1.5 The gRPC contract
This section formalizes the Model Service’s gRPC contract, grouping its operations into five concerns while separating the stable interface from changing implementation details.
- Five contract operation categories: Describe the five categories of Model Service operations and connect each category to the methods in the gRPC contract: response generation, model discovery, prompt management, custom-model registration, and embeddings.
Why it matters: The contract gives every platform service a consistent way to ask for model-related capabilities. - Embedding service responsibility: Explain what an embedding represents, why semantically similar text has nearby embeddings, and why embedding generation belongs in the Model Service rather than a separate service.
Why it matters: Centralizing embedding model calls lets the Data Service search organizational knowledge by meaning rather than just keywords. - Contract and implementation separation: Explain that the gRPC contract defines what the Model Service offers and how clients call it, not how providers, prompt storage, custom-model execution, or health checks are implemented; justify why a stable contract lets implementations change without breaking workflows.
Why it matters: A stable boundary allows the platform team to evolve infrastructure while preserving existing application integrations. - Chapter implementation scope: State that this chapter implements the core response-generation capability through provider abstraction, streaming, and operational concerns such as fallbacks and caching, while leaving implementations of discovery, prompt management, custom-model registration, and embeddings as backend-style patterns or reader exercises.
Why it matters: Knowing the scope distinguishes the chapter’s implemented model-serving path from contract operations intentionally left unimplemented.
Key terms: Model Service, gRPC contract, Chat, ChatStream, ListModels, GetModelCapabilities, embedding, vectors, Embed, ListEmbeddingModels
3.1.6 Request and response structures
This section defines the chat request and response data structures that precisely carry model inputs, advanced controls, generated outputs, and usage information through the Model Service.
- Chat request composition: Describe the fields of a ChatRequest and what information each supplies to the Model Service, including model selection, conversation messages, generation configuration, and named system prompts.
Why it matters: A clear request structure lets workflows ask different models for responses without each workflow inventing its own model API. - Advanced request controls: Explain how tool definitions and response_format extend a chat request, including why structured JSON output can be used directly by downstream systems.
Why it matters: These controls let an AI workflow produce machine-actionable data or request actions instead of only returning prose. - Chat response contents: Describe what a ChatResponse reports: generated content, the actual model and provider used, token usage, optional tool calls, and the reason generation finished.
Why it matters: The response gives a workflow both the result to handle and the operational facts needed to continue tool interactions and understand model behavior and cost. - Response metadata purposes: Explain why token usage, actual provider/model identity, tool calls, and finish reasons are returned as response metadata for cost tracking, workflow continuation, and diagnosing behavior.
Why it matters: Metadata turns a model result into something the platform can account for, react to, and troubleshoot rather than treating it as opaque text.
Key terms: ChatRequest, model, messages, Message, config, system_prompt_name, tools, ToolDefinition, FunctionDef, JSON Schema
3.2 Provider abstraction
This section introduces provider abstraction as the Model Service machinery that hides differences among AI providers behind one consistent interface.
- Multi-provider abstraction: Explain why the Model Service abstracts AI providers behind a consistent interface, despite providers performing similar model tasks in different ways.
Why it matters: A consistent interface lets applications use AI models without needing to handle each provider's unique behavior directly.
Key terms: Model Service, provider abstraction, multi-provider support, consistent interface, AI providers
3.2.1 How providers differ
This section shows how incompatible LLM provider APIs create lock-in and why the Model Service must hide those differences behind a unified abstraction.
- Provider API variations: Describe how major LLM providers differ in message formats, system-prompt handling, response structures, token-usage fields, and API conventions. Explain that these independently evolved differences obstruct a unified application interface.
Why it matters: Recognizing these differences explains why application code should not be written against one provider's API assumptions. - System prompt incompatibility: Explain the system-prompt incompatibility between OpenAI and Anthropic: OpenAI places a system prompt in the messages array with the system role, while Anthropic accepts it as a separate parameter. State that sending an OpenAI-shaped request to Anthropic without adaptation can cause the system prompt to disappear silently.
Why it matters: A missing system prompt can change application behavior without producing an obvious API error. - Provider lock-in costs: Explain why directly integrating application code with one provider creates vendor lock-in: adding a provider or fallback requires changing message construction, API calls, response parsing, and error handling. Describe how the Model Service abstraction replaces application rewrites with an adapter plus configuration changes, enabling fallback, provider experimentation, A/B tests, and data-driven model choices.
Why it matters: Provider independence lets teams maintain availability and choose models based on evidence rather than the cost of rewriting code.
Key terms: message formats, response structures, parameter names, API patterns, system prompts, token counting, system prompt, messages array, role: system, separate parameter
3.2.2 The unified provider interface
This section defines a unified provider interface and adapter architecture that lets applications use different AI model providers through the Model Service without provider-specific code.
- Provider adapter abstraction: Explain how the adapter pattern gives applications one unified interface while provider-specific adapters translate that interface to each provider's API. State that adding a provider requires a new adapter rather than changes to application code.
Why it matters: This abstraction keeps applications independent of the API differences among model providers. - Three-layer request architecture: Describe the three layers of the Model Service architecture: applications request a named model from the Model Service, the service routes according to configuration and applies user-defined fallbacks, and provider adapters handle provider API specifics.
Why it matters: Separating application calls, service routing, and provider translation lets the platform choose how requests are handled without exposing that implementation to applications. - Minimal provider contract: Describe the common ModelProvider contract as supporting synchronous chat, streaming chat, and capability reporting, with concrete provider classes inheriting from the abstract class and implementing provider-specific translation.
Why it matters: A deliberately minimal contract can work across providers while still supporting the core model operations the platform exposes. - Provider capability discovery: Explain why get_capabilities() is part of the provider interface: providers differ in features such as vision, function calling, and context-window size, and the Model Service uses reported capabilities for discovery operations.
Why it matters: Capability reporting allows the platform to describe what each provider can actually support instead of assuming all models offer the same features.
Key terms: adapter pattern, unified interface, provider adapters, Model Service, routing, fallback logic, configuration, ModelProvider, synchronous chat, streaming chat
3.2.3 OpenAI message format as platform standard
This section establishes the OpenAI message format as the Model Service’s canonical internal representation because it is a familiar de facto standard across providers and tools.
- Canonical message format: Explain that the platform represents a conversation as an array of message objects, each carrying a role and content, using the OpenAI message format as its canonical internal format.
Why it matters: A single message representation lets applications and platform services handle conversations consistently before adapters translate them for individual providers. - De facto standard rationale: Justify the choice of OpenAI’s format by explaining that it is a de facto industry standard, not a preference for OpenAI, and that this reduces translation for OpenAI and OpenAI-compatible vLLM while giving developers a familiar structure.
Why it matters: Choosing a widely recognized convention lowers integration friction while preserving the platform’s ability to support multiple providers. - Function-calling message fields: Describe how the standard message format represents function calling: an assistant message uses tool_calls when it wants to invoke functions, and a result message uses tool_call_id to report the corresponding result.
Why it matters: These fields preserve the connection between a model’s requested tool invocation and the result returned to the conversation.
Key terms: canonical representation, OpenAI message format, role, content, Model Service, de facto industry standard, OpenAI-compatible endpoint, vLLM, translation, provider
3.2.4 What adapters do
This section explains how provider adapters hide AI-provider differences by translating requests, responses, errors, and structured-output requirements into uniform platform behavior.
- Adapter translation responsibilities: Explain the five responsibilities of a provider adapter: message translation, parameter mapping, response normalization, error translation, and structured output translation. Describe how these responsibilities let the platform present one consistent model-service interface despite provider-specific APIs.
Why it matters: Adapters prevent workflows from needing provider-specific request formats, response parsing, and error-handling logic.
Key terms: adapter, message translation, parameter mapping, response normalization, error translation, ModelProviderError
3.2.5 The adapter implementation pattern
This section shows how provider adapters translate between the platform’s unified interface and each model provider’s API so hosted and self-hosted models can be used interchangeably.
- Adapter translation flow: Describe the common provider-adapter flow: accept unified platform input, translate it into a provider-specific request, call that provider’s API, and translate the provider response back into the unified format. Explain that the flow stays the same across providers even though each provider’s message and response structures differ.
Why it matters: This translation boundary lets application code use one model interface rather than handling every provider’s API format. - Provider-specific translation differences: Explain how provider adapters handle concrete format differences: Anthropic requires system-message extraction and content-block traversal, OpenAI keeps the system message in its message array and reads choices[0].message.content, and Google uses parts and candidates[0].content.
Why it matters: Knowing where formats differ clarifies why a unified interface requires provider-specific adapters rather than a single raw API call. - Self-hosted model equivalence: Explain why self-hosted models can use the same abstraction: vLLM, Text Generation Inference, and Ollama expose OpenAI-compatible APIs, so their adapters resemble the OpenAI adapter, with endpoint configuration and authentication as the main differences. State that this enables switching between cloud and local models without application code changes.
Why it matters: The abstraction allows routing or fallback between cloud providers and models running in your own datacenter without rewriting the application.
Key terms: adapter, unified interface, provider format, translation logic, system message extraction, content blocks, choices[0].message.content, parts format, candidates[0].content, self-hosted models
3.3 Streaming responses
Streaming delivers model output incrementally to improve perceived responsiveness and allow workflows to stop unsafe or unwanted generation early.
- Streaming user experience: Explain how streaming changes a model interaction from waiting for a completed response to seeing the response unfold as it is generated, and why early output improves perceived speed.
Why it matters: Users judge responsiveness heavily by when they first see useful output, not only by when generation finishes. - Early generation termination: Describe how streaming enables a workflow to inspect output as it arrives and abort generation early when it detects problematic content, avoiding the generation of tokens that will be rejected.
Why it matters: Early termination can prevent unnecessary generation once a workflow has identified output it should not accept.
Key terms: streaming, time-to-first-token, time-to-complete, early termination, abort
3.3.1 The streaming architecture
This section traces how streamed model tokens move from provider APIs through adapters, the Model Service, and the API Gateway to an SSE-connected client.
- End-to-end streaming pipeline: Trace the streaming path from a model provider to a frontend client: provider tokens are normalized by an adapter, streamed by the Model Service over gRPC to the API Gateway, and delivered by the gateway as Server-Sent Events.
Why it matters: Knowing each hop lets you locate latency or streaming failures in the layer responsible. - Provider stream normalization: Describe why provider adapters normalize differing provider streaming APIs into the platform representation before the Model Service forwards chunks. State examples of the differing formats: OpenAI uses delta.content chunks, Anthropic exposes a text_stream iterator through a context manager, and vLLM follows the OpenAI-compatible format.
Why it matters: Normalization keeps the rest of the platform independent of each provider's particular streaming API. - Client stream completion: Explain the external SSE delivery behavior: the frontend opens an SSE connection, receives pushed chunks continuously from the API Gateway, and treats a final chunk as the completion signal.
Why it matters: A client needs an explicit completion signal to know when a continuously delivered response has ended.
Key terms: provider adapter, Model Service, gRPC streaming, API Gateway, Server-Sent Events, SSE, platform representation, delta.content, text_stream, OpenAI-compatible
3.3.2 The ChatChunk message
This section defines the provider-independent ChatChunk message that gives streaming consumers a consistent way to append text, detect completion, identify models, and account for usage.
- Provider-independent streaming chunks: Explain how ChatChunk gives streaming consumers one common message format even though providers emit tokens in different native formats. Describe how the adapter layer normalizes provider-specific streaming events so workflow code does not need provider-specific conditional logic.
Why it matters: A uniform chunk format lets applications stream from different model providers without rewriting their response-handling code. - Chunk content and model identity: Describe the token and model fields in a ChatChunk. Explain that token text can be a word, part of a word, or punctuation and is normally appended as received, while the model field identifies the producer of each chunk, including chunks that may come from different models after a fallback.
Why it matters: Consumers need to render incremental text correctly and retain enough provenance to notice provider changes and attribute costs. - Stream completion and usage signals: Explain the final-chunk semantics of finish_reason and usage. State that finish_reason marks stream completion and can report stop, length, tool_calls, content_filter, or error, while final usage contains token counts only after generation completes, so streaming cost tracking occurs after the fact.
Why it matters: Correct completion handling prevents consumers from treating an unfinished stream as complete and sets the timing for reliable streaming cost accounting.
Key terms: ChatChunk, adapter layer, streaming responses, token, model, fallback chain, tokenization, finish_reason, usage, stop
3.3.3 Streaming and error handling
This section explains how the Model Service preserves partial streamed output when failures occur and why mid-stream failures restrict fallback recovery.
- Mid-stream error signaling: Describe how the Model Service reports a provider failure after streaming has started: it sends a final chunk whose finish_reason is set to error rather than attempting to return a normal error response.
Why it matters: A client needs an in-stream signal because it may already have displayed part of the model response. - Partial response preservation: Explain that output received before a mid-stream failure remains valid up to the error point, so clients can show the partial response together with an error indication rather than losing everything.
Why it matters: Preserving delivered content usually gives users a better experience than hiding useful partial output. - Streaming fallback limitations: Explain why a provider fallback generally applies only when failure happens before streaming begins: switching from a failed provider such as GPT-4o to another such as Claude mid-response cannot continue seamlessly because the response style may change jarringly.
Why it matters: This tradeoff helps teams set realistic reliability expectations for streamed model interactions.
Key terms: final chunk, finish_reason, error, mid-stream error, partial response, error indication, mid-stream errors, fallback behavior, synchronous requests, complete failures
3.4 Resilience: fallbacks and retries
This section introduces application-defined retry and fallback policies that the Model Service executes centrally to handle provider failures without workflow-level error handling.
- Resilience policy choices: Explain why model-provider outages, rate limits, and network timeouts require resilience policies, and describe how the appropriate retry or fallback behavior depends on the application’s availability and failure-tolerance needs.
Why it matters: Different AI applications need different responses to failure, so resilience cannot be a one-size-fits-all platform rule. - Platform-side policy execution: Describe the division of responsibility in resilience handling: the application configures retry, fallback, and error-handling policy, while the Model Service executes that policy without requiring the workflow to catch errors or implement retry loops.
Why it matters: Central execution keeps workflow code focused on application logic while applying its chosen resilience policy consistently.
Key terms: resilience primitives, retries, fallbacks, error handling, provider outages, rate limits, retry configuration, fallback configuration, Model Service, workflow
3.4.1 Retry configuration
This section explains how the Model Service configures retries for transient provider failures while avoiding wasteful retries for permanent errors.
- Retriable error classification: Distinguish transient errors that should be retried from permanent errors that should not. State that rate limits (HTTP 429), timeouts, and service-unavailable errors (HTTP 503) are retry candidates, whereas invalid requests (HTTP 400) are not.
Why it matters: Choosing the right retryable errors recovers from temporary provider problems without delaying failures that cannot succeed. - Retry timing configuration: Describe how max_retries, initial_delay, exponential_backoff, and max_delay control retry attempts and waiting time. Explain that exponential backoff lengthens successive waits, such as one, two, then four seconds, while max_delay caps that growth.
Why it matters: Bounded, increasing delays give a struggling provider time to recover without hammering it or making users wait indefinitely.
Key terms: transient failures, permanent failures, retry_on, rate limit errors, timeout errors, service unavailable errors, RetryConfig, max_retries, initial_delay, exponential_backoff
3.4.2 Fallback configuration
This section explains how fallback configuration moves failed model requests through an ordered provider chain while handling non-recoverable errors and recording the provider that ultimately responds.
- Ordered provider fallback chain: Describe how the providers field defines fallback providers in preference order after the primary provider fails, and explain that each provider receives its configured retry attempts before the service advances to the next provider.
Why it matters: An ordered fallback chain keeps model requests available while preserving your preferred provider choices. - Immediate-failure error policy: Explain how fail_on identifies errors that should stop the request immediately rather than trigger fallbacks, such as malformed requests and potentially content policy violations.
Why it matters: Failing immediately prevents pointless provider switching when the request itself is invalid or should not be accepted. - Fallback outcome attribution: State that a response succeeding through fallback identifies the provider that actually handled it, and explain why fallback events should be logged for reliability monitoring, cost attribution, and interpreting behavioral differences.
Why it matters: Recording the actual provider reveals hidden primary-provider problems and makes costs and response behavior understandable.
Key terms: FallbackConfig, providers, retry_config, fallback providers, fail_on, invalid request errors, content policy violations, provider field, logging, cost attribution
3.4.3 Configuration examples
This section shows how resilience configuration should vary for patient-facing, analytics, and cost-sensitive applications.
- Availability-focused resilience strategy: Describe the patient-facing configuration: aggressive short retries and a deep fallback chain prioritize continued responses over provider failures. Explain that cloud alternatives and a local Llama last resort preserve availability during cloud outages and can keep sensitive patient data on-premises.
Why it matters: A patient-facing application should fail over quickly rather than expose users to provider error messages. - Analytics pipeline failure handling: Explain why an analytics batch pipeline uses longer exponential-backoff retries with fallbacks disabled. Describe how surfaced failures let the pipeline decide whether to skip an item, queue it for later, or alert an operator.
Why it matters: Batch workloads can handle failed items deliberately instead of hiding problems behind automatic model fallbacks. - Cost-optimized fallback configuration: Describe the cost-sensitive strategy of retrying only timeouts and quickly falling back on rate limits to cheaper models such as GPT-4o-mini or a local model. Explain the trade-off: the application accepts potentially lower quality to reduce cost during high-traffic periods.
Why it matters: Fallbacks can manage spending and rate-limit pressure, not only recover from outages.
Key terms: aggressive retries, fallback chain, Claude, local Llama model, on-premises, exponential backoff, fallbacks disabled, rate limits, pipeline level, timeouts
3.5 Routing strategies
This section introduces routing as the Model Service’s pre-request provider-selection responsibility and explains why common routing policies belong centrally in the platform.
- Routing versus fallbacks: Distinguish routing from fallbacks by explaining that routing selects a provider before a request runs, whereas fallbacks handle provider failures after they occur.
Why it matters: This distinction clarifies whether the platform is proactively choosing a model or reactively recovering from a failure. - Configurable routing strategies: Describe the routing choices the Model Service can make when a workflow does not name a provider, including a fixed provider, cost-aware selection, load balancing, and capability-based selection. Explain that workflows can use built-in strategies, combine them, or supply custom routing logic.
Why it matters: Shared routing strategies let applications select suitable providers without every workflow rebuilding the same policy logic. - Centralized routing state: Explain why routing belongs in the Model Service: provider selection depends on cross-request state such as spending against a budget, active request counts, and provider capability matrices, which the service maintains and exposes to built-in or custom routing logic.
Why it matters: Centralized state enables routing decisions that an isolated workflow cannot make reliably from a single request alone.
Key terms: routing, fallbacks, provider selection, Model Service, routing strategies, cost-aware routing, load balancing, feature-based selection, custom routing logic, provider
3.5.1 Routing configuration
This section explains how application-defined routing configuration lets the Model Service choose a provider when a workflow does not explicitly name one.
- Routing configuration roles: Describe how RoutingConfig controls provider selection, including the strategy field, default_provider fallback, and per-strategy configuration fields.
Why it matters: These settings determine how the platform selects a model provider when an application leaves that choice open. - Routing ownership and bypass: Explain when provider routing runs and distinguish workflow-side routing preferences from platform-side routing execution and state management.
Why it matters: This boundary lets workflows express selection preferences while the Model Service makes decisions using shared spending and load state.
Key terms: RoutingConfig, strategy, default_provider, provider selection, application-side, workflows, Model Service, explicit provider, routing bypassed, spending state
3.5.2 Cost-aware routing
This section explains how the Model Service routes requests among models based on budget status and task complexity to control AI spending.
- Cost-aware routing decisions: Explain how cost-aware routing selects a provider using both current budget spending and the complexity of the task. Describe the decision pattern: use the cheapest option near the budget limit, inexpensive API models for routine work, and powerful expensive models only for complex work that justifies their cost.
Why it matters: This lets an application preserve model quality where it matters without allowing routine requests to exhaust its budget. - Automatic spending accounting: Describe how the Model Service maintains spending totals automatically from response token counts and provider pricing. State that budgets reset on a configurable schedule, typically monthly.
Why it matters: Routing can react to a budget only if the platform has an up-to-date measure of what model requests have cost. - Budget preservation behavior: Explain how shifting to low-cost local models near the end of a budget period preserves budget for high-value or emergency requests. Illustrate the pattern with routine work using cheap models and complex work using powerful models while funds remain.
Why it matters: The purpose is not merely lower average cost, but retaining limited spending capacity for requests where premium capability is most valuable.
Key terms: cost-aware routing, decision tree, budget limit, local models, API models, task complexity, token counts, provider pricing, running totals, budget resets
3.5.3 Load-based routing
This section explains how the Model Service routes requests to the least-loaded provider to increase throughput and avoid provider rate limits, while noting when output consistency makes such routing unsuitable.
- Least-loaded provider routing: Explain how load-based routing selects a provider for each incoming request. Describe how the Model Service tracks active requests per provider and sends each new request to the provider with the lowest current load.
Why it matters: This routing pattern spreads traffic proactively instead of waiting for a provider to fail or hit its limit. - Request counter lifecycle: Trace the lifecycle of an active-request counter during load-based routing. State that the counter increments before a request and decrements after it completes, whether the request succeeds or fails.
Why it matters: Accurate counters prevent failed requests from making a provider appear busy indefinitely. - Applicability and consistency tradeoff: Describe when load-based routing is useful and when it may be inappropriate. Explain that it helps use total capacity and avoid rate-limit-triggered fallbacks when providers have rate limits and similar latency, but different model families can produce inconsistent outputs; for consistency-sensitive work, balance across API keys for the same provider instead.
Why it matters: Choosing the right balancing scope lets a system gain capacity without sacrificing response consistency that an application requires.
Key terms: load-based routing, Model Service, active requests, least-loaded provider, counters, increment, decrement, rate limits, throughput, fallbacks
3.5.4 Feature-based routing
This section explains how feature-based routing selects only model providers that satisfy every capability required by a request and rejects unsupported requests early.
- Capability-based provider selection: Explain how feature-based routing chooses providers according to the capabilities a particular request requires, such as vision, function calling, large context windows, or specific output formats.
Why it matters: It prevents a request from being routed to a model that cannot perform the required kind of work. - Capability matrix intersection: Describe how the model service uses a capability matrix and set intersection to find providers that support all required features, with each requirement narrowing the eligible set.
Why it matters: The intersection rule ensures that the selected provider can satisfy the complete request rather than only one of its requirements. - Fail-fast unsupported requests: Explain why the router fails fast with a clear error when no provider supports a required feature or when no provider supports the full combination of features.
Why it matters: Failing early gives a clear cause instead of producing confusing failures from an incapable model.
Key terms: feature-based routing, capabilities, providers, vision, function calling, context windows, capability matrix, set intersection, required features, capable providers
3.5.5 Combining patterns
This section explains how the Model Service combines feature, cost, and load-routing strategies into an ordered decision process.
- Combined routing sequence: Describe how combined routing evaluates feature requirements first, cost logic second, and load balancing last when selecting a model provider. Explain that each stage narrows the remaining provider options.
Why it matters: This ordering lets the platform choose an appropriate provider automatically without asking application code to implement routing rules. - Capability-first routing priority: Explain why feature requirements are hard constraints in combined routing, while cost preferences are soft constraints that cannot override required model capabilities.
Why it matters: A lower-cost model is useful only if it can actually perform the requested task. - Transparent application routing: Describe how application code can make a normal platform.models.chat(messages) call while the Model Service transparently applies the combined routing policy. Illustrate how routine requests can shift to a local model near a budget limit while complex cases or feature-specific requests retain suitable cloud providers.
Why it matters: Keeping routing inside the platform gives applications simple model calls while policy decisions remain centralized and consistent.
Key terms: combined routing strategy, feature requirements, hard constraints, cost logic, soft preferences, load balancing, cost preferences, vision-capable providers, platform.models.chat(messages), Model Service
3.6 Rate limiting
This section explains how platform-level rate limiting protects an organization from runaway model usage and cost while distinguishing policy enforcement from provider throttling.
- Platform-level rate limiting: Explain why the Model Service enforces rate limits before requests reach providers, rather than leaving limiting to providers or individual applications. Describe how a platform-wide view protects against cross-application abuse and runaway spending.
Why it matters: Central enforcement prevents one workflow, user, or attack from consuming the organization's shared model budget. - Limit scopes and consumption dimensions: Describe the per-user, per-workflow, and global scopes for rate limits, and distinguish request-per-minute limits from token-per-minute limits. Explain why both request and token limits may be needed to prevent automated abuse and expensive oversized contexts.
Why it matters: Different scopes and measurements prevent both unfair capacity monopolization and high-cost usage patterns that request counts miss. - Rate-limit error handling: Distinguish platform rate-limit responses from provider-originated rate limits in the retry and fallback flow. Describe the Retry-After indicator and explain why platform limits must not trigger a fallback to another provider.
Why it matters: Treating an organizational spending cap like a provider outage would bypass the cap and amplify the very cost problem it is meant to stop. - Rate-limit telemetry and tuning: Describe how tracking which workflows hit rate limits, how often, and when turns limit events into capacity-planning evidence. Explain why consistently hitting limits should prompt investigation of caching, batching, unnecessary calls, architecture, or policy settings rather than an automatic increase.
Why it matters: Limit-hit data helps teams improve capacity and workflow efficiency before excessive demand becomes an unexpected bill.
Key terms: Model Service, platform-level rate limiting, provider rate limits, fallback chains, scope, per-user limits, per-workflow limits, global limits, requests per minute, tokens per minute
3.7 Caching for cost and performance
Introduces caching as a cost- and latency-reduction technique for LLM applications that requires choosing among multiple strategies and trade-offs.
- Multi-level caching trade-offs: Explain that caching can reduce both API costs and response latency in LLM applications, but is not a single technique. Describe that multiple caching strategies operate at different levels and must be chosen according to their distinct trade-offs.
Why it matters: Choosing an appropriate caching strategy can substantially improve application speed while lowering model API spending.
Key terms: caching, LLM applications, caching strategies, trade-offs, API costs, latency
3.7.1 Two levels of caching
This section distinguishes application-level response caching from provider-level prompt caching and explains how exact and semantic response caching can work together to reduce latency and cost.
- Response caching strategies: Describe how response caching stores and returns complete model responses, and distinguish exact-match caching from semantic caching. Explain why an application can check an exact match first and then use semantic similarity after an exact-match miss.
Why it matters: Choosing the right response-cache strategy can eliminate model calls for repeated questions while still serving paraphrased requests. - Response cache benefits and limits: Explain the benefits and constraints of response caching, including its zero model-call cost and sub-millisecond latency for identical requests, its requirement that messages and configuration match, and its conflict with intentionally varied model outputs.
Why it matters: This helps you decide when reusing a completed answer improves the application and when it damages the intended user experience. - Provider prompt caching: Distinguish provider-level prompt caching from response caching. Describe how providers reuse previously seen prompt content to discount cached tokens while the application still makes the API call, and explain how prompt caching complements response caching after a response-cache miss.
Why it matters: Provider prompt caching can substantially reduce spend for repeated prompt context even when the application cannot reuse a complete answer.
Key terms: response caching, exact-match caching, semantic caching, embedding, similarity threshold, cache hit, identical requests, zero cost, sub-millisecond latency, varied responses
3.7.2 The cache interface
This subsection defines configurable response and prompt caching in the Model Service, including cache-key correctness, eviction behavior, and provider-specific prompt-cache handling.
- Response cache key composition: Explain how the Model Service generates response-cache keys from the model, messages, and configuration, and why a changed temperature must produce a different key even when the messages are identical.
Why it matters: Correct cache keys prevent the platform from returning a response produced under different model settings. - Response cache lifecycle controls: Describe how response caching is optional and configurable per request, and how the cache removes entries through LRU eviction at capacity and TTL expiration when entries become stale.
Why it matters: These controls let applications gain caching benefits without retaining stale responses or allowing the cache to grow without bound. - Provider-level prompt caching: Distinguish provider-level prompt caching from response caching, and describe how the adapter layer handles provider-specific controls while applications choose per request whether to enable that caching.
Why it matters: Provider differences should not leak into application logic, while applications still need control over whether to use provider-level caching.
Key terms: response caching, cache keys, hash, model, messages, configuration, per request, LRU eviction, capacity, TTL expiration
3.7.3 Monitoring cache effectiveness
This section explains how the Model Service measures whether response and prompt caches save money and uses those measurements to tune caching and attribute costs accurately.
- Response cache hit-rate monitoring: Explain what response cache hit rate measures and how it guides response-cache configuration. Describe how a much lower-than-expected hit rate can indicate variable queries, an overly short TTL, or excessive temperature.
Why it matters: A cache reduces latency and cost only when enough requests actually reuse cached responses. - Cache-aware cost attribution: Describe how prompt-cache savings are obtained from provider response metadata and aggregated across requests. Distinguish the cost of a response-cache hit, a prompt-cache hit, and a full cache miss, and explain why the service must record the path each request took.
Why it matters: Without identifying the cache path per request, reported AI spending can overstate savings or assign costs incorrectly.
Key terms: response cache hit rate, TTL, temperature, cache configuration, prompt cache savings, response metadata, response cache hit, prompt cache hit, full cache miss, discounted rate
3.8 Observability: Cost tracking and metrics
This section introduces how the Model Service records per-request metrics and sends them to the Observability Service so AI spending can be traced and optimized.
- Per-request cost observability: Explain that the Model Service tracks comprehensive metrics for every model request and feeds them to the platform's Observability Service. Describe how this makes provider costs traceable to the workflows, features, or users responsible for token consumption.
Why it matters: Per-request telemetry turns an unexplained AI-provider bill into information teams can use to find spending drivers and optimize them.
Key terms: Model Service, comprehensive metrics, Observability Service, token consumption, workflows
3.8.1 What the model service tracks
This section defines the per-request metrics the Model Service records to measure model usage, actual execution, performance, failures, and attribution.
- Token usage metrics: Describe the token metrics recorded for each Model Service response, including prompt, completion, total, response-cache, and prompt-cache tokens. Explain how these metrics support cost calculations because providers charge per token and caches can eliminate or reduce that cost.
Why it matters: Token-level metrics reveal what requests cost and how much caching is saving. - Actual provider and model: Explain why metrics record the provider and specific model that actually handled a request rather than only the model originally requested. Relate this to fallback behavior and accurate cost attribution across providers with different rates.
Why it matters: A requested model is not always the model that served the request, so actual execution data is necessary for trustworthy operations and billing analysis. - Latency and outcome tracking: Describe the timing and outcome metrics for a request: time to first token, total duration, time spent retrying or attempting fallbacks, and success or categorized failure. Explain how first-token latency reflects streaming user experience, total duration supports throughput and timeout planning, and error categories reveal recurring versus transient problems.
Why it matters: These measurements let teams diagnose slow experiences and recognize whether failures are isolated events or systemic patterns. - Workflow and user attribution: Explain how workflow_id and user_id attach Model Service metrics to the workflow and user that initiated a request. Describe the later analysis this enables, including identifying workflows with high token use or error rates and users driving high cost.
Why it matters: Attribution turns aggregate model metrics into actionable information about which applications and users create cost or reliability problems.
Key terms: prompt tokens, completion tokens, total tokens, response cache, prompt caching, provider, model, fallbacks, cost attribution, time to first token
3.8.2 Feeding the observability service
The Model Service publishes per-request metrics to the Observability Service so teams can attribute AI costs, analyze usage, and detect operational problems in real time.
- Observability metric publication: Describe how the Model Service records metrics locally on every request and publishes them to the platform Observability Service for cross-platform aggregation.
Why it matters: Publishing model telemetry centrally makes model behavior visible alongside the rest of the platform. - Multi-dimensional usage analysis: Explain that the Observability Service aggregates Model Service data by provider, model, workflow, user, and time, enabling teams to compare spending, effectiveness, resource consumption, and usage trends.
Why it matters: These dimensions let teams locate where AI usage and cost originate instead of seeing only a single total. - Cost attribution and decisions: Explain how observed token usage and model choices are attributed back to workflows and features, so teams can identify the cause of a budget increase and decide whether higher model quality justifies higher cost.
Why it matters: Cost attribution turns aggregate AI spend into evidence for product and engineering trade-offs. - Real-time operational visibility: Describe how real-time dashboards expose request rates, error rates, and latency percentiles so teams can monitor launches and detect deployment problems within minutes.
Why it matters: Fast visibility lets teams respond to a failing launch before users must report the problem.
Key terms: Model Service, Observability Service, metrics, per-request metrics, provider, model, workflow, user, time, cost-effectiveness
3.8.3 Enabling informed decisions
This section shows how Model Service observability data enables teams to attribute costs, diagnose problems, detect anomalies, and make evidence-based optimization decisions.
- Cost driver analysis: Explain how token usage by workflow and model, plus cost per user, identify where AI spending comes from and support targeted cost optimization. Describe the example of using a cheaper model for short document summaries while reserving GPT-4 for complex documents.
Why it matters: Knowing which features, models, and users create costs lets teams reduce spending without blindly degrading capability. - Performance bottleneck diagnosis: Describe how response-time metrics reveal latency regressions and how drilling into the data can connect a slowdown to frequent fallbacks caused by primary-provider rate limiting. Explain why load-based routing across multiple providers can address that particular bottleneck.
Why it matters: Performance metrics let teams find and correct a user-visible slowdown before users complain. - Anomaly detection alerts: Explain how changes in error rates, token usage per request, and cache hit rates can expose problems early, including provider failures, prompt bloat, and less predictable queries. Describe why alerting enables proactive response instead of reactive firefighting.
Why it matters: Alerts turn gradual or sudden metric changes into issues teams can address before they become larger incidents or costs. - Evidence-based model routing: Explain how comparative quality, cost, and error-rate data can guide model and provider selection instead of guesswork. Describe the example where Claude Sonnet can handle classification when it has similar quality to GPT-4 at 60% lower cost, while provider-specific errors can also change routing.
Why it matters: Using observed outcomes allows teams to choose models that meet quality needs at lower cost and with greater reliability. - Request-level cost accountability: State how built-in observability eliminates the mysterious-invoice problem by accounting for every token, attributing every request, and tracing every cost to the feature that incurred it. Explain that this gives budget owners evidence to explain, defend, or optimize spending.
Why it matters: Clear accountability makes AI budget changes understandable and actionable for both engineering and business owners.
Key terms: token usage by workflow, token usage by model, cost per user, targeted optimization, GPT-4o-mini, GPT-4, average response time, fallbacks, rate-limiting, primary provider
3.9 Integrating with the SDK
This section positions the SDK's ModelClient as the layer that turns developers' simple model calls into gRPC requests routed through the API Gateway to the Model Service.
- SDK-to-model-service translation: Explain how the SDK lets a developer call platform.models.chat(messages) without interacting with gRPC directly, translating Python method calls into Protocol Buffer messages for the Model Service.
Why it matters: This translation gives workflow developers a simple Python interface while the platform handles service communication. - Gateway request path: Trace the request path from developer code through GenAIPlatform and ModelClient to a gRPC channel at the API Gateway, which routes the request to the appropriate backend service.
Why it matters: Knowing the path clarifies which layer owns client communication and which layer routes requests to backend services.
Key terms: SDK, ModelClient, Protocol Buffer messages, gRPC, Model Service, platform.models.chat(messages), GenAIPlatform, API Gateway, gRPC channel, client object
3.9.1 The ModelClient
This subsection explains how the SDK creates a ModelClient that routes model requests through the Gateway and wraps generated gRPC stubs in a Python-friendly interface.
- ModelClient responsibilities: Describe the three responsibilities of a ModelClient: connecting to the Gateway, constructing Protocol Buffer requests for model operations, and converting Protocol Buffer responses into Python objects usable by application code.
Why it matters: The ModelClient is the SDK layer that makes the remote model service practical to call from Python workflows. - Gateway routing configuration: Explain how service_name="models" configures routing so that the API Gateway forwards ModelClient requests to the model service.
Why it matters: Routing by service name lets the SDK target the correct platform service through the shared Gateway. - Generated gRPC stub: Describe the generated ModelServiceStub as code produced when the .proto contract is compiled, providing Python methods for gRPC operations such as Chat, ChatStream, and ListModels while handling serialization and network communication.
Why it matters: The generated stub supplies the low-level gRPC connection that the SDK wraps into a convenient developer interface.
Key terms: ModelClient, Protocol Buffer requests, Protocol Buffer responses, Python objects, service_name="models", routing, API Gateway, model service, _stub, ModelServiceStub
3.9.2 Method implementation pattern
This section traces how ModelClient.chat translates a Python call into a routed gRPC request and returns a Python-friendly response.
- Three-phase client method pattern: Describe the three phases shared by SDK client methods: converting Python inputs to Protocol Buffer messages, making the gRPC call with routing metadata, and converting the Protocol Buffer response back into a Python dataclass. Explain that this hides Protocol Buffer details from workflow code.
Why it matters: This pattern lets application developers use ordinary Python objects while the SDK handles service communication. - Default model routing behavior: Explain that an empty model name allows the Model Service to select a provider using its routing logic, such as cost-aware or load-based routing. State that the response identifies the model that actually generated the response, even when the caller did not name one.
Why it matters: Callers can delegate model selection to platform policy while still seeing which model served each request.
Key terms: ModelClient, Protocol Buffer, ChatMessage, ChatResponse, _dict_to_proto_msg, _stub.Chat(), model name, routing logic, cost-aware routing, load-based routing
3.9.3 Streaming support
This section explains how the SDK exposes Model Service streaming as a Python generator that progressively yields response chunks to workflow code.
- SDK streaming generator: Describe how calling platform.models.chat_stream returns a generator whose iterations progressively yield streamed response chunks as Python objects. Explain that each iteration waits for the next chunk rather than returning one complete response.
Why it matters: This lets a workflow show generated output as it arrives instead of making users wait for the model's full response. - Streaming delivery path: Trace the streaming path from provider to caller: the Model Service sends chunks as the provider produces them, the Gateway forwards them, and the SDK pulls them from the gRPC stream and yields them to application code.
Why it matters: Knowing this path clarifies that the SDK hides network-stream handling while preserving progressive delivery.
Key terms: platform.models.chat_stream, generator, chunks, gRPC stream, ModelClient, Model Service, Gateway, provider
3.9.4 The complete picture
This section traces a ModelClient chat request from Python workflow code through the platform and provider, showing how the platform hides distributed integration complexity.
- End-to-end chat request flow: Trace a chat request from Sarah's workflow code through the SDK, Gateway, Model Service, OpenAI adapter, and back to a Python response. Include the key request transformations and routing decisions described in the complete flow.
Why it matters: This flow shows how one simple SDK call is carried safely across the platform and an external model provider. - Platform-managed prompt and telemetry: Describe the Model Service work performed after it receives the routed request: selecting an adapter from the requested model, looking up and prepending the registered system prompt by name, recording metrics, and returning the response over gRPC.
Why it matters: These centralized steps let application code request a model and named prompt while the platform applies shared behavior and captures operational data. - Intent-focused SDK abstraction: Explain the developer experience produced by this stack: Sarah makes one readable method call and receives a Python object, while the platform hides serialization, gRPC channels, Gateway routing, provider adaptation, and external API calls.
Why it matters: Hiding infrastructure mechanics lets workflow authors express application intent without reimplementing integration logic.
Key terms: GenAIPlatform, ModelClient, ChatRequest, Protocol Buffer, gRPC, Gateway, requested model, OpenAI adapter, system_prompt_name, registered prompt
3.10 Summary
This summary consolidates how the Model Service provides a provider-independent model gateway with streaming, resilience, routing, cost controls, observability, and SDK access.
Where this is used later
- The simplest assistant: a model call in a workflow (chapter 10) · The simplest assistant invokes the Model Service to generate a response.
- Cost attribution and budget tracking (chapter 7) · Cost attribution and budget tracking rely on accurate model-usage data.
- Target lifecycle and evaluation (chapter 8) · Target lifecycle and evaluation can retrieve and compare managed prompt versions.
- The prompt that makes it all work: context engineering (chapter 10) · Context engineering uses prompts as part of the assistant's managed behavior.
- Offline evaluation: comparing variants before deployment (chapter 8) · Offline evaluation can compare prompt versions on the same questions before a change is deployed.
- Service contract and complete retrieval flow (chapter 5) · The Data Service’s retrieval flow needs embedding operations to make documents searchable.
- From documents to searchable knowledge (chapter 5) · Searchable organizational knowledge relies on embeddings to represent document meaning.
- Generating embeddings (chapter 5) · The ingestion pipeline calls for generating embeddings from document chunks.
Concepts explained in this chapter
- model serviceA model service is a platform service that provides a unified interface to multiple AI model providers while handling provider-specific APIs, routing, resilience, and operational concerns.
- token countToken usage is the quantity of language-model input and output tokens consumed by an AI operation.
- adapterThe adapter pattern is a design pattern that translates one interface into another interface expected by clients.
- interfaceA consistent interface is a uniform software interface that presents the same interaction model across different implementations or providers.
- cachingResponse caching is the storage and reuse of complete model responses for requests that match according to a defined request-matching rule.
- fallbackA fallback is a failure-handling strategy that switches to an alternative provider or execution path when the primary one fails.
- stream processingA streaming response is an output delivery mode that sends generated content incrementally as it becomes available instead of waiting for the complete result.
- retryA retry is a repeated attempt to perform an operation after a transient failure or unsuccessful attempt.
- routingRouting is the selection of a model provider for a request according to a configured strategy and runtime conditions.
- prompt cachingPrefix caching is a caching method that reuses computations for an identical leading portion of a prompt.
- model discoveryModel discovery is the retrieval of available AI models and their supported capabilities from a model provider or service.
- model registrationCustom model registration is the process of declaring a custom model so that a model service can identify and invoke it.
- protobufProtocol Buffers is a language-neutral, platform-neutral serialization format for structured data and service interface definitions.
- finish reasonA finish reason is a response field indicating why a model stopped generating output, such as natural completion, a limit, a tool call, filtering, or an error.
- ChatChunkChatChunk is a message structure representing one incremental fragment of a model's streamed response, including the fragment text, producing model, completion status, and optional usage information.
- output formatA model response constrained to a specified machine-readable structure rather than unconstrained text.
- load balancingLoad-based routing distributes requests among providers or execution paths according to their current workload or capacity.
- exponential backoffExponential backoff is a retry timing strategy that increases the delay between successive attempts, typically by a multiplicative factor.
- rate limitRate limiting is the controlled restriction of how many requests or tokens a system accepts from a client, workflow, or organization over a period of time.
- embeddingAn embedding is a numerical vector representation of content that preserves aspects of its meaning so semantically similar content has nearby representations.
- messageA message is a structured conversational item containing a role and content, optionally accompanied by metadata for tool interactions.
- promptAn instruction or input supplied to an AI model to elicit a response or behavior.
- gRPCgRPC is a high-performance remote procedure call framework that uses service definitions and serialized messages for communication between software components.
- fallback configurationFallback configuration defines alternative providers, their retry settings, and failure conditions for execution after an earlier provider path is unavailable or exhausted.
- contractAn API is a defined interface through which software components communicate and invoke capabilities, including interfaces exposed by external systems.
- OpenAI message formatThe OpenAI message format is a message representation in which a conversation is an array of objects containing fields such as role and content, with additional fields for tool invocation results.
- observabilityThe ability to infer a system's internal behavior and health from its emitted telemetry and operational records.
- tool definitionToolDefinition is the example request structure that describes a function an AI model may call, including its function metadata and parameter schema.
- tool usetool_calls is a message field containing an assistant's requested tool or function invocations.
- gRPC contractA gRPC contract is a formal service interface that defines the operations and interaction structure exposed between gRPC services.
- abstractionModel abstraction is an interface design that hides provider-specific model details so applications can work with interchangeable models.
- routing metadataRouting metadata is request-associated key-value information that identifies the backend service to which a gateway should direct a call.
- vendor lock-inVendor lock-in is dependence on a provider's technology or interfaces that makes switching to alternatives difficult or costly.
- dialogueA conversation is a structured sequence of messages exchanged between a user, an assistant, and the system instructions governing the interaction.
- model selectionModel selection is the process of choosing an AI model for a task based on criteria such as quality, cost, latency, or reliability.
- cost attributionCost attribution is the assignment of incurred service or infrastructure costs to users, applications, activities, or other accounting categories.
- observability serviceA service that collects and exposes operational information about application behavior, including latency, usage, and performance.
- tokenA token is a unit into which text is divided for processing and usage measurement by an AI model.
- context windowA context window is the bounded amount of input and prior interaction content that a language model can process for one generation.
- vectorizationAn embedding model is an AI model that converts content into numerical vector representations capturing semantic or learned relationships.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
