liveBook · Manning

Latency budget

As explained in Quantization and Fast Inference

A latency budget is a maximum allowable elapsed time for a computation or service operation.

A latency budget sets the maximum acceptable elapsed time for an operation. On edge devices, set it alongside a power budget: inference costs energy, and sustained workloads can hit a thermal knee that pushes latency above early measurements. section 11.0

Sustained edge testing

MacBook Air M3, iPhone 16, and Pixel 10 Pro serve as reference devices. Latency measured in the first 30 seconds may not reflect performance after five minutes of sustained work. section 11.0

How Quantization and Fast Inference covers it

How deep it goes
Introduces latency and power budgets and sustained thermal effects, but specifies no numerical limits or allocation procedure.
Examples
Names MacBook Air M3, iPhone 16, and Pixel 10 Pro as reference devices and contrasts early measurements with sustained workloads.
What it assumes
Assumes experience shipping quantized server models while targeting engineers new to phone deployment.
What you can do after
Evaluate edge inference latency and power against achievable budgets under sustained workloads.
How it connects
Connects latency budgets with power budgets, thermal behavior, quantized inference, and mobile runtimes.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.