liveBook · Manning

Thermal knee

As explained in Quantization and Fast Inference

A thermal knee is the point during sustained computation at which thermal management causes a sharp reduction in performance.

A thermal knee is the point during sustained computation when thermal management sharply reduces performance. It makes cold-start latency a poor proxy for the latency users experience after a device has been working for minutes. section 11.1.2 section 11.0

Mobile SoCs use junction-temperature protection, with a TJMax around 95–105 °C, and a virtual-skin-temperature loop that protects the user's hand. Under tight inference loops, the skin-temperature loop can trigger first and sharply lower clocks, halving throughput within seconds. Whether the knee appears depends on the workload and device; a short benchmark can miss it. section 11.1.2 section 12.2

Measure sustained behavior rather than relying on a cold-start result:

  • Run a sustained loop and compare throughput in the last 30 seconds with the first 30 seconds; report the retention percentage alongside the workload and duration. section 11.1.2
  • Treat the result as workload-specific. A 300-second M3 vision run lost 35.6% over a 60-second sweep, while Pixel results stayed within 1.3 percentage points of 100% retention. section 11.1.2
  • A light M3 vision workload showed 106.47% retention after 300 seconds, partly because Core ML dispatch warm-up slowed the initial window; that result does not rule out throttling under heavier conditions. section 12.2
Figure 11.2 Thermal retention across modalities and devices. Bar height is 100 × last_30s / first_30s of sustained throughput; 100% (dashed line) is the no-throttle baseline. Bar hatches encodes modality (vision plain, audio dotted, LLM cross-hatch). Annotations report retention percentage and first → last absolute throughput in the modality's native unit. The M3 vision 60 s bar loses 35.6%; the M3 audio 300 s bar shows a 54.1% gain; the Pixel bars hold within 1.3 percentage points of 100%.

Compare throughput retention across modalities and devices, including the M3 vision drop and Pixel's near-flat results. section 11.1.2

How Quantization and Fast Inference covers it

How deep it goes
Explains thermal-control loops and sustained-throughput measurement, but does not provide a general predictive model for when throttling begins.
Examples
Compares M3 vision throughput loss, near-flat Pixel retention, and a light M3 run that gains throughput after warm-up.
What it assumes
Assumes familiarity with ML inference deployment and introduces the phone's system-on-chip as a CPU, GPU, and ML accelerator.
What you can do after
Diagnose sustained-load degradation by comparing first- and last-window throughput under a stated workload and duration.
How it connects
Connects thermal throttling with sustained throughput, cold-start latency, and power budgets.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.