What you will learn in Designing AI Systems
A guide to production-ready platforms
Chapter by chapter
Chapter 1: Why your AI projects need a platform
This chapter introduces why moving AI prototypes into production requires shared platform infrastructure rather than one-off application code.
What you learn in chapter 1 →Chapter 2: Building with the platform: SDK and API design
This chapter introduces the SDK and API contracts that turn the platform’s services into productive developer and consumer interfaces.
What you learn in chapter 2 →Chapter 3: The Model Service: Your platform’s gateway to AI models
This introduction positions the Model Service as the platform component that turns application requests into AI responses while hiding provider and operational complexity.
What you learn in chapter 3 →Chapter 4: The Session Service: Teaching your AI to remember
This chapter introduces the Session Service as the platform component that preserves conversational context, stores it as structured session data, and prepares it within model context-window limits.
What you learn in chapter 4 →Chapter 5: The Data Service: teaching AI what your organization knows
This chapter introduces the Data Service as platform infrastructure that grounds AI responses in searchable organizational knowledge rather than relying only on conversational memory.
What you learn in chapter 5 →Chapter 6: Tools and Guardrails: Enabling safe, managed AI actions
This introduction explains why AI actions need platform-managed tools and guardrails, and outlines the services that provide them.
What you learn in chapter 6 →Chapter 7: The Observability Service: seeing what your AI does
This introduction frames the Observability Service as the platform component that unifies telemetry across services so teams can understand AI behavior, cost, performance, and quality.
What you learn in chapter 7 →Chapter 8: The Experimentation Service: improving what your AI does
This chapter introduction positions the Experimentation Service as the infrastructure that turns observability signals into a continuous, controlled loop for improving AI behavior.
What you learn in chapter 8 →Chapter 9: The Workflow Service: orchestrating and deploying AI applications
This introduction positions the Workflow Service as the platform component that turns developer workflow code into scalable HTTP applications by separating deployment operations from request-time execution.
What you learn in chapter 9 →Chapter 10: Building an AI assistant: putting the platform to work
This introduction frames the chapter as an incremental build of Claw, a complete AI assistant that composes every platform service while keeping application code focused on assistant behavior.
What you learn in chapter 10 →Concepts explained in this book
- model serviceA model service is a platform service that provides a unified interface to multiple AI model providers while handling provider-specific APIs, routing, resilience, and operational concerns.
- guardrailA guardrail is an execution policy that constrains AI system inputs, actions, and outputs according to safety, business, permission, or regulatory requirements.
- spanA span is a distributed-tracing record representing one timed operation within a trace and its relationship to parent and child operations.
- trajectoryA trace is a correlated record of a request's journey through a system, including its operations, timing, metadata, and nested work units across components.
- tool usetool_calls is a message field containing an assistant's requested tool or function invocations.
- session serviceA session service is a platform service that stores and manages conversational state, user preferences, interactions, and conversation history across sessions.
- stream processingA streaming response is an output delivery mode that sends generated content incrementally as it becomes available instead of waiting for the complete result.
- workflow serviceA workflow service is a platform service that coordinates sequential and parallel operations, state management, retries, and failure recovery in complex processes.
- retryA retry is a repeated attempt to perform an operation after a transient failure or unsuccessful attempt.
- observability serviceA service that collects and exposes operational information about application behavior, including latency, usage, and performance.
- qualityA score is a numerical or categorical value representing how well an observed operation or output performed against an evaluation criterion.
- memoryModel-managed memory is a capability in which an AI model selects, stores, and retrieves information from persistent memory outside its immediate context window.
- generationA generation is an observability record that wraps a span and adds model, provider, token, cost, cache, and first-token timing information for a model call.
- experimentation-as-a-service platformAn experimentation service is a platform service that provides infrastructure for running controlled experiments and measuring the effects of application changes.
- data serviceA data service is a reusable platform service that ingests documents, generates vector embeddings, and supports semantic search over organizational knowledge.
- context engineeringContext engineering is the deliberate assembly of the information supplied to a model call within constraints such as token capacity and attention behavior.
- tool serviceA platform service that registers and discovers external tools, APIs, and services while handling authentication, rate limiting, recovery, and caching.
- API gatewayAn API gateway is a gateway component that provides a unified entry point for accessing backend services and applies shared cross-cutting controls.
- chunkingFixed-size chunking divides text into chunks containing a configured number of tokens with optional overlap between consecutive chunks.
- workflow runtime serverA workflow runtime server is SDK library code that imports a workflow module, registers an HTTP route, receives requests, invokes a workflow function, and returns its responses as a service.
- sessionA session is a structured record representing one conversation between a user and an AI application, including its messages and identifying metadata.
- similarity searchVector search retrieves items by comparing numerical vector representations of queries and candidate content.
- workflowA workflow is a defined sequence of dependent operations that coordinates processing steps and manages their execution state.
- ObservabilityClientObservabilityClient is an SDK client that buffers spans, logs, and metrics locally and flushes them to the Observability Service in batches.
- messageA message is a structured conversational item containing a role and content, optionally accompanied by metadata for tool interactions.
- retrieval indexA retrieval index is a searchable data structure that organizes document representations and associated metadata so relevant information can be found efficiently.
- text chunkA document chunk is a segment of an ingested document, with associated source text or metadata, stored as an independently searchable unit.
- platformA software and infrastructure foundation that provides shared capabilities for building and operating applications.
- software development kitA software development kit is a collection of programming tools, libraries, documentation, and related resources for building applications against a software platform or service.
- dialogueA conversation is a structured sequence of messages exchanged between a user, an assistant, and the system instructions governing the interaction.
- server-sent eventsServer-Sent Events is an HTTP-based mechanism for continuously delivering unidirectional, incrementally generated events from a server to a client.
- synchronous callSynchronous execution is an execution model in which the caller waits for an operation to complete before receiving its result.
- interfaceA consistent interface is a uniform software interface that presents the same interaction model across different implementations or providers.
- observabilityThe ability to infer a system's internal behavior and health from its emitted telemetry and operational records.
- contractAn API is a defined interface through which software components communicate and invoke capabilities, including interfaces exposed by external systems.
- ReActAn agent loop is an iterative execution pattern in which a model selects tools, observes their results, and continues reasoning until it produces a final response or reaches a stopping condition.
- HTTPHTTP is the stateless application-layer protocol used to transfer requests and responses between networked clients and services.
- multi-step orchestrationMulti-step orchestration is the coordination of a sequence of dependent AI workflow steps with step-specific capabilities, error handling, and recovery.
- data ingestionDocument ingestion is the process of importing a document, extracting and chunking its content, generating representations, and storing the resulting data for retrieval.
- embeddingAn embedding is a numerical vector representation of content that preserves aspects of its meaning so semantically similar content has nearby representations.
- workflow decoratorA workflow decorator is a function annotation that converts an ordinary function into a managed workflow process.
- vector databaseA vector database is a database system optimized for storing, indexing, and retrieving vector representations.
- schemaA database schema is the formal organization of a database's tables, columns, relationships, and constraints.
- adapterThe adapter pattern is a design pattern that translates one interface into another interface expected by clients.
- TracedServiceTracedService is the book's base service class that automatically captures tracing context and handles telemetry-related service operations.
- hybrid searchHybrid retrieval combines multiple retrieval methods, such as semantic vector search and keyword search, into a unified result ranking.
- response modeA response mode is a workflow configuration that determines whether results are returned synchronously or incrementally as a stream.
- gRPCgRPC is a high-performance remote procedure call framework that uses service definitions and serialized messages for communication between software components.
- conversational aiA personal AI assistant is an AI application that helps an individual by understanding requests, providing information, and performing actions on the user's behalf.
- metadata filterA metadata filter is a query condition that restricts candidate records according to descriptive key-value attributes.
- storage interfaceA storage interface is an abstract contract that defines operations for persisting and retrieving application data independently of a backend.
- protobufProtocol Buffers is a language-neutral, platform-neutral serialization format for structured data and service interface definitions.
- policy enforcementPolicy enforcement is the application of rules that constrain system inputs, outputs, or actions to comply with organizational requirements.
- human-in-the-loopHuman review is the inspection and validation of an item by a person before it is accepted for an intended purpose.
- routingRouting is the selection of a model provider for a request according to a configured strategy and runtime conditions.
- AI agentAn agent is an AI system composed of a model and execution harness that can select and perform actions toward a goal.
- fallbackA fallback is a failure-handling strategy that switches to an alternative provider or execution path when the primary one fails.
- MCPMCP, or the Model Context Protocol, is a standard protocol for connecting AI applications to servers that expose tools and other contextual capabilities.
- asynchronous requestAn asynchronous request is an interaction in which the client submits work, receives an acknowledgment or job identifier, and retrieves progress or results later.
- OpenAI message formatThe OpenAI message format is a message representation in which a conversation is an array of objects containing fields such as role and content, with additional fields for tool invocation results.
- tool registryA tool registry is a managed catalog that stores tool definitions and versions, supports discovery, and governs their registration and lifecycle.
- readiness checkA readiness probe is an automated check that determines whether a service instance is initialized and ready to receive traffic.
- validationInput validation is the process of checking incoming requests or data against defined validity, safety, and policy requirements before processing.
- token countToken usage is the quantity of language-model input and output tokens consumed by an AI operation.
- organizational knowledgeOrganizational knowledge is the body of policies, procedures, guides, directories, and other information maintained by an organization.
- gRPC contractA gRPC contract is a formal service interface that defines the operations and interaction structure exposed between gRPC services.
- vectorizationAn embedding model is an AI model that converts content into numerical vector representations capturing semantic or learned relationships.
- cachingResponse caching is the storage and reuse of complete model responses for requests that match according to a defined request-matching rule.
- pollingPolling is the repeated checking of an endpoint or resource until a desired state or result is available.
- workflow compositionWorkflow composition is the pattern of building an application workflow by invoking other independently deployed workflows instead of duplicating their logic.
- job recordA job record is the Workflow Service's stored representation of one submitted workflow execution, including its status, payloads, progress, result, error, and assigned container.
- job idA job ID is a unique identifier assigned to an asynchronous workflow execution so clients and services can refer to its job record.
- context variableA context variable is a dynamically scoped variable whose value can differ across execution contexts such as threads or asynchronous tasks.
- nlp-based chunkingSemantic chunking detects topic boundaries by comparing embeddings of consecutive sentences and starting chunks where semantic similarity drops sharply.
- session managementSession management is the creation, persistence, retrieval, and expiration of state associated with user or application sessions.
- pgvectorpgvector is a PostgreSQL extension that provides vector data types, similarity operators, and indexes for storing and querying embeddings.
- retrieval-augmented generationA retrieval-augmented generation pipeline retrieves relevant context and supplies it to a generative model when producing a response.
- scalable infrastructure foundationShared infrastructure and organizational capability that supports consistent development and operation of applications.
- structure-aware chunkingStructure-aware chunking is the division of document text into chunks while preserving or using structural boundaries such as headings and sections.
- credential isolationCredential isolation is a security technique that keeps credentials outside application code and limits their exposure to a protected execution boundary.
- telemetryOperational data emitted by a software system to represent its behavior and health, such as logs, metrics, traces, and generation records.
- promptAn instruction or input supplied to an AI model to elicit a response or behavior.
- A/B testA controlled experiment that compares alternative variants by assigning them to different user or traffic groups and measuring outcomes.
- content filteringThe detection or suppression of inappropriate, harmful, or otherwise disallowed content in model inputs or outputs.
- atomicityAtomicity is the database transaction property that requires all operations in a transaction to succeed together or none to take effect.
- deployment pipelineA deployment pipeline is an automated sequence of steps that builds, packages, and releases software for execution in a target environment.
- serializationSerialization is the conversion of in-memory data structures into a transferable or storable representation.
- continuous improvementAn asynchronous improvement loop evaluates completed interactions after serving them and feeds the resulting scores into systematic system improvement.
- parsingParsing is the analysis of structured or semi-structured input to identify and extract its meaningful components.
- RecursiveCharacterTextSplitterRecursive splitting divides text by trying a hierarchy of natural separators and recursively using finer separators when pieces exceed the configured chunk size.
- reciprocal rank fusionReciprocal rank fusion is a rank-aggregation algorithm that combines result lists using the reciprocal positions of items rather than their scores.
- abstractionModel abstraction is an interface design that hides provider-specific model details so applications can work with interchangeable models.
- automated scoringAn automated scorer is a non-generative evaluation procedure that computes response-quality scores from recorded metrics or rules.
- retrieval-augmented memoryRetrieval-augmented memory retrieves relevant information from prior sessions and incorporates it into a current interaction.
- contextual aiContext-aware intelligence is the ability of an AI application to generate behavior using relevant external knowledge and conversational context.
- lexical searchKeyword search retrieves content by matching literal terms or tokens in a query against indexed content.
- interceptorA gRPC interceptor is a client- or server-side component that intercepts remote procedure calls to apply behavior before or after invocation.
- developer experienceDeveloper experience is the quality and usability of the workflows, interfaces, and tools that developers use to build and operate software.
- rollbackA rollback is the restoration of a deployed system to a previously known version or configuration after a problematic change.
- prompt cachingPrefix caching is a caching method that reuses computations for an identical leading portion of a prompt.
- load balancingLoad-based routing distributes requests among providers or execution paths according to their current workload or capacity.
- platform serviceA reusable infrastructure or software service provided centrally to support multiple applications or teams.
- histogramA histogram metric records observations in value ranges so their distribution and percentile statistics can be calculated.
- service-oriented architectureService-oriented architecture is an architectural pattern that organizes software as independently focused services communicating through defined interfaces.
- truncationTruncation is a context-management technique that removes older or excess messages so the remaining content fits within a token budget.
- database transactionA database transaction is a sequence of database operations treated as one unit of work with a shared commit or rollback outcome.
- lazy initializationLazy initialization is the creation of an object or resource only when it is first needed.
- vendor lock-inVendor lock-in is dependence on a provider's technology or interfaces that makes switching to alternatives difficult or costly.
- budget alertA budget alert is a notification triggered when tracked or projected spending crosses a defined budget threshold.
- offline evaluationOffline evaluation assesses an AI system or change using previously collected datasets rather than live interactions.
- ivfflatIVFFlat is an approximate vector index that partitions vectors into clusters and searches selected clusters to accelerate similarity queries.
- index configurationAn index configuration is the set of immutable and adjustable parameters that define an index's identity, embedding model, and document chunking behavior.
- exponential backoffExponential backoff is a retry timing strategy that increases the delay between successive attempts, typically by a multiplicative factor.
- liveness checkA liveness probe is a health-check interface that indicates whether a running service instance is alive and able to be managed by infrastructure.
- control planeA control plane is the part of a distributed system responsible for deployment, configuration, operations, and management rather than handling application requests.
- data planeA data plane is the part of a distributed system that processes application requests and carries out runtime operations.
- tool registrationTool registration is the declaration of an AI-accessible tool together with its invocation details, credentials, and behavioral metadata.
- sliding windowA sliding window is a context-management technique that retains the most recent items while storing older items outside the active processing window.
- deploy CLIThe deploy CLI is a command-line tool that packages workflow code, publishes its container image, and registers the workflow for deployment.
- dimensionDimensionality is the number of scalar values composing a vector representation.
- configuration managementConfiguration management is the controlled handling of application settings, secrets, and environment-specific values across system operation.
- routing metadataRouting metadata is request-associated key-value information that identifies the backend service to which a gateway should direct a call.
- tool definitionToolDefinition is the example request structure that describes a function an AI model may call, including its function metadata and parameter schema.
- context windowA context window is the bounded amount of input and prior interaction content that a language model can process for one generation.
- recoveryCrash recovery is the detection and re-dispatch of work whose executing process or container has unexpectedly disappeared.
- credential storeA credential store is a protected repository that stores secrets and applies access controls, encryption, and lifecycle policies to them.
- message roleA message role is a field that identifies whether a conversation message contains system instructions, user input, or an assistant response.
- online evaluationOnline evaluation assesses an AI system or change using live production interactions and their outcomes.
- execution policyA domain-specific rule set that determines whether an AI-generated action may be executed under the current business context and workflow state.
- on delete cascadeON DELETE CASCADE is a relational database constraint action that automatically deletes dependent rows when their referenced row is deleted.
- database indexA database index is a persisted auxiliary structure that accelerates retrieval of rows matching query conditions.
- concurrencyConcurrency is the management of multiple in-progress operations so that their execution can overlap without requiring simultaneous CPU execution.
- correlation idA correlation ID is an identifier carried across related records or service calls so they can be associated with the same request or operation.
- child workflowA child workflow is a workflow invoked by another workflow and executed as an independently managed deployment.
- hierarchical memoryHierarchical memory is a memory architecture that organizes information into layers with different levels of structure, compression, recency, and retention priority.
- monitoringThe observation of service telemetry to assess the operational condition of a platform or system.
- audit trailAn audit trail is a chronological record of system events and actions that supports accountability, investigation, and compliance reporting.
- annotation queueAn annotation queue is an ordered collection of items awaiting human labeling or review.
- model registrationCustom model registration is the process of declaring a custom model so that a model service can identify and invoke it.
- rate limitRate limiting is the controlled restriction of how many requests or tokens a system accepts from a client, workflow, or organization over a period of time.
- UvicornUvicorn is an ASGI server that listens for network connections and dispatches incoming requests to an ASGI application.
- service discoveryService discovery is the mechanism by which distributed components locate available services and their connection endpoints.
- model discoveryModel discovery is the retrieval of available AI models and their supported capabilities from a model provider or service.
- trace contextDistributed trace context is propagation metadata that associates work performed across services with the same distributed trace.
- agentic RAGAgentic RAG is a retrieval-augmented generation approach in which a model decides when and how to retrieve information and can iterate on searches.
- retrieve-then-generate pipelinePipeline-based RAG is a retrieval-augmented generation approach in which retrieval runs as a fixed mechanical step for each request.
- foreign keyA foreign key is a database field or set of fields that references a key in another table to enforce referential integrity.
- jsonbJSONB is PostgreSQL's binary JSON data type for storing structured JSON values with database indexing and querying support.
- PII scrubbingPII redaction is the removal or masking of personally identifiable information from generated or transmitted content.
- relevanceA retrieval relevance distribution is the statistical distribution of relevance scores produced for retrieved results across queries.
- batchingBatching is the accumulation of multiple items so they can be processed or transmitted together.
- similarity metricA similarity score is a numerical value representing how closely two items or vector representations match, with larger values indicating greater similarity.
- prompt injectionPrompt injection is an attack that manipulates model instructions or supplied content to induce behavior that conflicts with intended system instructions or policies.
- dimensional labelA metric label is a named attribute attached to a metric that enables grouping and querying by dimensions such as provider, model, or workflow.
- time seriesA time series is an ordered sequence of timestamped values showing how measurements change over time.
- defense in depthDefense in depth is a security pattern that applies multiple independent protective controls so that failure of one control does not eliminate protection.
- versioningThe systematic identification and preservation of successive versions of software, data, models, or configuration.
- logThe recording of application or system events, diagnostics, and operational information.
- LLM-as-a-judgeAn LLM judge is a language model used to evaluate another model's output against specified criteria.
- service coordinationService coordination is the management of dependencies and interactions among components of a service-based system.
- JSONJSON is a text-based data interchange format for representing structured values such as objects, arrays, strings, numbers, and nested data.
- isolationIsolation is the separation of software workloads so that their execution, resources, and interactions are constrained from affecting one another.
- storage abstractionA storage abstraction is an interface or design boundary that separates data-storage operations from the underlying persistence technology.
- call_parallelcall_parallel is a workflow API operation that invokes multiple child workflows concurrently and returns their results in submission order.
- counterA counter metric is a monotonically accumulating measurement of events or quantities over time.
- ON CONFLICT clauseThe ON CONFLICT clause is a PostgreSQL SQL construct that specifies how an insert should proceed when it violates a uniqueness or exclusion constraint.
- psycopgpsycopg2 is a Python adapter library for connecting to and communicating with PostgreSQL databases.
- idempotencyAn idempotent operation has the same externally observable effect when repeated, whereas a non-idempotent call can create an additional effect each time it runs.
- cosine distanceCosine distance measures dissimilarity between vectors based on the angle between their directions, with zero indicating identical direction.
- HNSWHNSW is a graph-based approximate nearest-neighbor index that organizes vectors into layered networks of nearby connections.
- evaluationEvaluation is the systematic assessment of an AI system or change against defined evidence and criteria.
- output formatA model response constrained to a specified machine-readable structure rather than unconstrained text.
- autoscalingAuto-scaling is the automatic adjustment of computing resources in response to workload demand.
- MCP serverAn MCP server is a server that exposes tools through the Model Context Protocol for discovery and invocation by compatible clients.
- model selectionModel selection is the process of choosing an AI model for a task based on criteria such as quality, cost, latency, or reliability.
- lost in the middleThe lost in the middle phenomenon is a language-model behavior in which information positioned in the middle of a long context receives less attention than information near the beginning or end.
- asynchronous programmingAsynchronous execution is an execution model in which a request returns before its operation completes, allowing processing to continue independently in the background.
- evaluation datasetAn evaluation dataset is a collection of test cases containing inputs, expected outputs, and metadata for assessing an AI system or change.
- knowledge baseA knowledge base is an organized repository of domain-specific facts, documents, or policies that can provide authoritative information to an application.
- format detectionFormat detection is the process of determining the file format of an input document before selecting a parser.
- finish reasonA finish reason is a response field indicating why a model stopped generating output, such as natural completion, a limit, a tool call, filtering, or an error.
- protocThe Protocol Buffer compiler is a code-generation tool that transforms Protocol Buffer service and message definitions into language-specific source code.
- stubA stub is a generated client-side proxy that represents remote operations as local callable methods.
- proto fileA proto file is a Protocol Buffers definition file that declares structured messages and service operations for code generation.
- ChatChunkChatChunk is a message structure representing one incremental fragment of a model's streamed response, including the fragment text, producing model, completion status, and optional usage information.
- tool executionTool execution is the invocation of an externally defined function, API, or service on behalf of an AI application.
- rolling summaryA rolling summary is a continuously updated compressed representation of older conversation content that preserves its essential narrative.
- experiment targetAn experiment target is a record that references a versioned prompt, model configuration, or retrieval configuration and tracks its evaluation and lifecycle metadata.
- retrieval rankingA rank is the ordinal position of an item in an ordered list of search results.
- trace_generationtrace_generation is a TracedService context-manager operation that creates and manages a generation trace primitive for a language-model call.
- trace_operationtrace_operation is a context-manager method that records a non-LLM service operation as a tracing span and yields context for nested calls.
- target lifecycleA target lifecycle is the progression of an experiment target through draft, testing, active, and deprecated states based on evaluation outcomes.
- inspection point modelA policy-enforcement pattern that places guardrail evaluations at defined locations in an AI request, tool, external-system, and response flow.
- policy checkThe checking of action parameters against business rules and domain constraints rather than only structural or type requirements.
- PostgreSQLPostgreSQL is an open-source object-relational database management system.
- api sprawlAPI sprawl is the uncontrolled proliferation of separate API interfaces with fragmented conventions and access patterns.
- fallback configurationFallback configuration defines alternative providers, their retry settings, and failure conditions for execution after an earlier provider path is unavailable or exhausted.
- parent workflowA parent workflow is a workflow that invokes and coordinates one or more other workflows.
- tool resultA tool result is the structured output returned by an invoked tool and appended to the conversation for subsequent model reasoning.
- trace treeA span tree is the hierarchical structure that represents nesting and parent-child relationships among spans in a trace.
- replace-on-reingestReplace-on-reingest updates indexed content by deleting data for an existing document and rebuilding it from the newly ingested version.
- technical debtTechnical debt is the future cost of maintaining or changing a system because expedient implementation choices created structural limitations or rework.
- health checkA health check is an automated test that determines whether a running service instance is functioning and able to receive work.
- output guardAn output guardrail is a post-generation inspection mechanism that detects, transforms, blocks, or routes unsafe or invalid model responses before delivery.
- groundingGrounding is the process of connecting an AI application's responses to authoritative external or organizational knowledge.
- managed toolA managed tool is an externally invocable capability governed by a platform through registered identity, lifecycle, ownership, credentials, and operational controls.
- labelA ground truth label is a reference judgment assigned to an example output, typically by human annotation, for evaluating other scoring methods.
- independent deploymentAn independently deployable service is a service that can be released, scaled, and versioned without coordinating deployment with other services.
- circuit breaker patternA circuit breaker is a resilience pattern that temporarily stops calls to a repeatedly failing dependency and later tests whether it has recovered.
- time-series databaseA time-series database stores measurements indexed by time and supports time-range queries over metrics.
- health check endpointA health endpoint is an API endpoint that reports whether a service process is running or ready to receive traffic.
- token limitA token limit is a configuration constraint specifying the maximum number of model tokens permitted for an input, output, or request.
- cost attributionCost attribution is the assignment of incurred service or infrastructure costs to users, applications, activities, or other accounting categories.
- aggregate metricAn aggregate metric is a system-level measurement computed by summarizing observations across multiple events, requests, or components.
- asyncio.to_threadasyncio.to_thread is a Python API that runs a synchronous callable in a separate thread while allowing the calling coroutine to remain asynchronous.
- summarizationSummarization is the generation of a shorter representation that preserves the salient information from a longer text or interaction.
- scalabilityThe ability of a system to maintain acceptable operation as workload or demand increases.
- experimentationAn experiment is a controlled procedure that assigns alternatives to traffic and measures their outcomes.
- ai sprawlAI sprawl is the exponential growth of disconnected AI services and overlapping infrastructure implementations across an organization.
- container imageA container image is an immutable package containing application code, runtime components, libraries, and dependencies for creating containers.
- deployment modelDeployment configuration is the set of settings that specifies how a workflow is exposed, scaled, and run on a platform.
- AI-specific metricA quantitative measurement describing an AI application's usage, quality, or conversational behavior.
- safety validationSafety validation is the systematic checking of inputs, outputs, and intermediate operations against safety, privacy, and policy requirements.
- agent orchestrationThe coordination of agent steps, tools, models, and state to complete a multi-step task.
- tokenA token is a unit into which text is divided for processing and usage measurement by an AI model.
- containerA container is a standardized executable unit that packages an application with its dependencies and isolates its processes while sharing the host operating system kernel.
- performanceA quantitative measurement of a system's operational behavior or task results.
- stateActual state is the observed condition of a deployed workload and its running instances.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
