liveBook · Manning

Capacity planning

As explained in AI Model Evaluation

Capacity planning is the process of determining the computational resources required to operate a system under expected and peak workloads.

Capacity planning determines whether an AI-powered feature can serve expected and peak workloads reliably, quickly, and cost-effectively. It connects infrastructure decisions to latency, traffic volume, operating cost, and product experience. section 4.0

A practical capacity plan evaluates:

  • CPU, memory, and GPU requirements at different traffic levels, using load testing to measure how resource needs scale. section 4.5.1
  • Cost-performance trade-offs, such as whether lower latency from a larger instance justifies its additional expense. section 4.5.1
  • Vertical scaling versus horizontal scaling, based on whether the model benefits more from larger machines or distributed smaller instances. section 4.5.1
  • Bottlenecks beyond traffic volume, including slow inference, feature retrieval, poor batching, cold starts, memory leaks, GPU saturation, and dependency limits. section 4.5.1

Autoscaling tools such as Kubernetes autoscaling, AWS Auto Scaling, and GCP managed instance groups can adjust resources to traffic patterns, but they do not replace performance evaluation. Continue measuring bottlenecks at the model, service, dependency, and product-experience layers. section 4.5.1

Shadow traffic

Shadow testing duplicates production requests, so the evaluated system may impose at least twice the computational load. Capacity planning must account for this overhead without degrading production. Stateful systems also require isolated data stores or safeguards against duplicate writes, while agent systems may need dry-run adapters, sandboxed APIs, read-only credentials, recorded responses, or mock tool outputs. section 4.3.3

Figure 4.2 illustrates how shadow traffic adds additional load to the supporting infrastructure such as feature database layer, metadata services, caches and user context lookups. This is one of the primary risks of shadow traffic that's worth over communicating so that the machine learning or platform infrastructure teams aren’t surprised by an increase in requests as a result of the shadow traffic evaluation.

The figure traces how shadow traffic increases load across feature databases, metadata services, caches, and user-context lookups. section 4.3.3

How AI Model Evaluation covers it

How deep it goes
Explains resource allocation, load testing, scaling choices, cost trade-offs, and infrastructure bottlenecks.
Examples
Uses shadow traffic to demonstrate duplicate computational load, stateful side effects, and supporting-infrastructure strain.
What it assumes
Assumes familiarity with production models, infrastructure resources, traffic load, and service dependencies.
What you can do after
Evaluate infrastructure requirements, choose scaling strategies, and diagnose bottlenecks before production deployment.
How it connects
Connects load testing, autoscaling, vertical scaling, horizontal scaling, latency, and shadow traffic.

Also explained in

Compare how 4 books explain capacity planning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.