liveBook · Manning

Throughput

As explained in Delivering Machine Learning Projects: From design to deployment

Throughput is the amount of work or data that a system processes within a specified time interval.

Throughput measures how many model executions or service requests a system completes within a defined period. It differs from latency, which measures the time for one result. section 8.2.7 section 9.3

Evaluate throughput as a production requirement by checking:

  • Test with hardware, workloads, and serving conditions that mirror production; isolated measurements can be misleading. section 8.2.7
  • Account for cold starts, parallel execution, downstream serving bottlenecks, and hardware-specific behavior rather than assuming linear scaling. section 8.2.7
  • Compare measured throughput with end-to-end performance targets during production-system review. section 9.3

Parallel execution

A model that executes one episode per second might achieve throughput near nine episodes per second when ten models run in parallel, even though individual latency remains unchanged. section 8.2.7

Table 9.3 Nonfunctional requirements in an ML-powered system (view table figure)

Requirement

Description

Notes

$Cost

How much it costs to execute the system.

Need to consider long life but cheap infrastructure versus short life but more expensive.

Environmental cost

The amount of environmental damage that can be accepted per use of the service.

Thought should be given to environmental impacts including greenhouse gas emissions and the use of noxious chemicals in air conditioning and cooling systems. Be cognizant of the consumption of metals, such as rare earth and gold, which may be involved in fielding your service in some domains.

Latency/wall clock

The amount of elapsed time from a service request to a delivered result.

In some applications, this is considered in terms of a flat requirement such as less than 0.5 s; alternatively, present as an expectation around a distribution or range of outcomes.

Throughput

The number of requests that can be served in a given amount of time.

This defines how many customers utilizing the services can be supported simultaneously by the system.

Queue policy

Preference for serving excess requests.

If there are more requests than can be supported by the system throughput, or a request cannot be served in the required clock time, what should the behavior of the system be? Should strict FIFO/LIFO behavior be enforced? Are there other requirements to prioritize certain requests?

Failure policy

When a request cannot be served without breaching a nonfunctional requirement, what should happen?

Some systems should fail silently, some should provide an exception/failure response and message, and some should serve the request outside of the nonfunctional requirement. In some cases, it’s even appropriate to close the system after failure.

Durability

How long the service is to perform without failure.

Some systems become unreliable over time. Determining how long the models must work in service dictates the kind of measures that are required to mitigate this.

Reliability

How many failures (breach of nonfunctional requirements) the system can tolerate.

Consider all sources of failure together as well as looking at them one at a time.

Use the nonfunctional-requirements table to place throughput alongside latency, cost, and environmental impact when reviewing an ML-powered system. section 9.3

How Delivering Machine Learning Projects: From design to deployment covers it

How deep it goes
Explains throughput, contrasts it with latency, and covers cold starts, parallelism, bottlenecks, and production-like measurement.
Examples
Uses cold starts, ten parallel models, production-mirroring test harnesses, and service-request targets as evaluation examples.
What you can do after
Evaluate throughput against end-to-end targets using production-like hardware, workloads, and serving conditions.
How it connects
Connects throughput with latency, parallelism, model serving, nonfunctional requirements, and end-to-end performance.
Cover of Delivering Machine Learning Projects: From design to deployment
liveBookmanning.com
Read Delivering Machine Learning Projects: From design to deployment on liveBook Get the book at manning.com

Also explained in

Compare how 20 books explain throughput →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Delivering Machine Learning Projects: From design to deployment. The section links point to the parts of the book it draws on.