liveBook · Manning

Network throttling

As explained in AI Model Evaluation

Throttling is the deliberate limitation of request or traffic processing rates to control resource use.

Network throttling deliberately limits shadow-traffic volume so infrastructure costs and production risk remain manageable while evaluation still gathers useful evidence. section 4.3.3

Apply throttling against explicit evaluation goals:

  • Replicate only a subset of real traffic rather than sending every request to the shadow system. section 4.3.3
  • Choose enough traffic to achieve statistical significance, cover important user segments, or exercise selected edge cases. section 4.3.3
  • Combine throttling with smart sampling and capacity planning, because duplicate requests can strain production resources. section 4.3.3

A cost-aware shadow run

A team evaluates a new inference system using a sampled portion of live requests, throttles that stream, and runs shadow traffic on cheaper CPUs when GPU capacity is unnecessary. It validates that the reduced volume still covers target segments and edge cases. section 4.3.3

Figure 4.2 illustrates how shadow traffic adds additional load to the supporting infrastructure such as feature database layer, metadata services, caches and user context lookups. This is one of the primary risks of shadow traffic that's worth over communicating so that the machine learning or platform infrastructure teams aren’t surprised by an increase in requests as a result of the shadow traffic evaluation.

Inspect how shadow traffic increases load across feature databases, metadata services, caches, and user-context lookups. section 4.3.3

How AI Model Evaluation covers it

How deep it goes
Explains sampling decisions, statistical coverage, hardware trade-offs, and capacity risks but does not specify a throttling algorithm.
Examples
Uses sampled real traffic, cheaper CPUs, statistical significance, user segments, and edge cases as evaluation choices.
What it assumes
Assumes familiarity with shadow traffic, production infrastructure, and evaluation goals such as coverage or statistical significance.
What you can do after
Operate/configure shadow evaluations by selecting traffic volume and hardware according to coverage goals and resource limits.
How it connects
Connects shadow traffic with smart sampling, capacity planning, infrastructure costs, and production performance.

Also explained in

Compare how 4 books explain network throttling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.