liveBook · Manning

Thermal throttling

As explained in Quantization and Fast Inference

Skin-temperature throttling reduces processor operating frequency or capacity when device-surface temperature reaches a thermal limit.

Thermal throttling is a workload-dependent loss of sustained inference throughput when thermal limits constrain device operation. Short-burst latency can hide it; steady-state throughput is the more relevant measure for sustained workloads. section 11.3.4

Measured outcomes vary with the workload:

  • On a fanless M3 MacBook Air, EfficientNet-Lite0 with Core ML palettize_4bit on the ANE fell from about 2,660 to 2,360 images per second over five minutes, an 11% soft-throttle. section 11.3.4
  • Llama-3.2-1B INT8 decode on Pixel stayed near 23.8 tokens per second, consistent with a memory-bandwidth ceiling that did not move with temperature. section 11.3.4
  • A 300-second M3 vision loop ended faster than it began: the light CPU-heavy workload did not reach its thermal envelope, while Core ML warm-up slowed the early measurements. section 12.2

For sustained workloads, compare throughput in the first and last 30 seconds of a thermally representative run. Interpret the result alongside workload and device conditions; a flat or rising rate does not show that heavier workloads will avoid throttling. section 11.3.4 section 12.2

How Quantization and Fast Inference covers it

How deep it goes
Compares first- and last-30-second rates in sustained runs, but does not detail thermal-control internals.
Examples
Compares M3 EfficientNet throughput loss, Pixel Llama decode stability, and an M3 vision loop that warmed up without throttling.
What you can do after
Diagnose sustained-load regressions by comparing first- and last-30-second throughput in a representative soak.
How it connects
Connects thermal behavior to inference throughput, Core ML warm-up, workload placement, and memory bandwidth.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.