liveBook · Manning

Technical debt

As explained in Designing AI Systems

Technical debt is the future cost of maintaining or changing a system because expedient implementation choices created structural limitations or rework.

Technical debt is the future cost of maintaining or changing a system when expedient implementation choices create structural limitations or rework. In AI organizations, it often grows from uncoordinated progress: individually sensible model, storage, and architecture decisions become a fragile ecosystem of one-off solutions and hidden assumptions. section 1.1.4 section 1.2.2

AI technical debt accumulates when:

  • Effective prompts, safety configurations, and other solutions remain trapped inside individual applications instead of becoming reusable capabilities. section 1.1.4
  • Teams repeat experiments and rebuild infrastructure for unstructured data, multi-step reasoning, external tools, and real-time generation. section 1.2.2
  • Every change must navigate disconnected services, hidden assumptions, and infrastructure assembled independently by different teams. section 1.1.4

From chatbot prototype to production liability

A developer can demonstrate a promising chatbot in days, but real users expose the need for performance monitoring, cost controls, safety guardrails, and operational infrastructure. Without shared platform services, the prototype becomes a months-long engineering project that other teams cannot reuse. section 1.0 section 1.2.2

Figure 1.2 The GenAI Infrastructure Explosion. The model API call now represents just 2% of total system complexity—even less than in traditional ML. Modern GenAI applications require vastly expanded infrastructure: knowledge management systems, safety and compliance layers, session management for conversations, tool integration platforms, advanced observability, and multi-provider model orchestration. This architectural shift explains why simple prototypes become complex production systems.

Use the infrastructure-explosion figure to compare the small model API call with the broader production system: knowledge management, safety and compliance, session management, tool integration, observability, and model orchestration. section 1.2.2

How Designing AI Systems covers it

How deep it goes
Explains organizational causes and infrastructure consequences, but stops short of detailing a remediation implementation.
Examples
Contrasts a chatbot prototype with production demands and shows a minimal model call beside expanded GenAI infrastructure.
What you can do after
Evaluate why AI features become harder to build, deploy, and maintain when shared platform capabilities remain absent.
How it connects
Connects technical debt to AI sprawl, platform foundations, operational challenges, infrastructure, and shared services.

Also explained in

Compare how 16 books explain technical debt →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.