liveBook · Manning

Scalable infrastructure foundation

As explained in Designing AI Systems

Shared infrastructure and organizational capability that supports consistent development and operation of applications.

A scalable infrastructure foundation provides shared capabilities for developing and operating applications as demand and organizational complexity grow. It addresses the gap between a prototype that works in a demo and a production system that must support users, changes, and multiple teams. section 1.3 section 1.2.2

Without a shared foundation, teams can make sensible local choices yet accumulate disconnected services and hidden assumptions. New features then require more infrastructure work, and changes become harder to predict. This is AI sprawl: uncoordinated growth that undermines organizational agility. section 1.1.4 section 1.2.2

A production foundation needs capabilities that grow with the system:

  • Configuration management secures secrets and environment-specific settings. section 1.3
  • Resource allocation scales individual components according to load. section 1.3
  • Service coordination manages dependencies between platform components. section 1.3
  • Deployment automation enables reliable updates without downtime. section 1.3

For generative AI, shared services may also provide knowledge ingestion, context assembly, session memory, agent orchestration, and AI-specific governance. These capabilities support reuse across applications rather than requiring each team to rebuild its own infrastructure. section 1.2.2

From chatbot prototype to production

A simple chatbot prototype crashed at 50 concurrent users because it lacked production infrastructure. Shared scaling, configuration, coordination, and deployment capabilities make that operational burden less visible to application developers. section 1.3

Figure 1.2 The GenAI Infrastructure Explosion. The model API call now represents just 2% of total system complexity—even less than in traditional ML. Modern GenAI applications require vastly expanded infrastructure: knowledge management systems, safety and compliance layers, session management for conversations, tool integration platforms, advanced observability, and multi-provider model orchestration. This architectural shift explains why simple prototypes become complex production systems.

The diagram contrasts the small model-call component with the broader infrastructure needed for a production generative AI application. section 1.2.2

How Designing AI Systems covers it

How deep it goes
Identifies production infrastructure requirements and their rationale, then defers implementation to a layered, service-oriented architecture.
Examples
Contrasts disconnected AI services with a chatbot prototype that crashes at 50 concurrent users.
What you can do after
Evaluate production infrastructure needs against prototype limitations and shared-service requirements.
How it connects
Connects infrastructure scaling with configuration management, resource allocation, service coordination, deployment automation, and model abstraction.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.