Power is the rate at which a device uses electrical energy, measured in watts. For inference, it is a deployment constraint as well as a cost: a configuration that is fast when cold can exceed a device’s thermal budget and slow down under sustained load. section 1.1 section 11.1
Power is distinct from energy per operation or per token. Moving data can cost much more energy than arithmetic: an FP32 fused multiply-add costs roughly 3–5 pJ, versus 0.2–0.5 pJ for INT8. These are order-of-magnitude estimates; exact costs vary by hardware. A 7B-parameter model moves roughly 15 GB through memory and consumes nearly 2 J per generated token. section 1.1 section 1.4
Figure 1.1 Fetching data from HBM costs roughly 1,700× more energy than an INT8 multiply-add. This gap explains why inference systems are bottlenecked by memory, not compute.

The energy comparison shows why reducing memory movement can matter more than accelerating arithmetic. section 1.1
Reducing precision from 16-bit floats to 8-bit or 4-bit integers cuts memory footprint, bandwidth, and energy roughly in proportion to the bit reduction. Quantization also introduces approximation error, so the deployment decision must balance efficiency against the representation’s range and resolution. section 1.4
Set a latency-and-power budget before tuning. Filter configurations by time to first useful output, sustained throughput retention under thermal load, and mean power. A phone’s thermal budget is closer to 4–6 W; continuous work above that ceiling can trigger throttling within minutes. A laptop on AC has a different constraint. section 11.1
Same workload, different power budgets
For EfficientNet-Lite0, an M3’s best configuration uses 4-bit palettization on the Neural Engine: 0.36 ms at 6.3 W. A Pixel’s best uses INT8 on XNNPACK: 4.09 ms at 1.75 W. The M3 is about 11× faster, but its power draw is unsuitable for the phone budget described above. section 11.1
Measure sustained behavior, not just a cold-start latency. One on-device vision run timed 200 images after 20 warm-up frames, then ran a 300-second sustained loop with power sampled across a 90-second overlap. Its end-to-end p95 was 9.00 ms; stage timing showed that JPEG decode, not the model forward pass, dominated the mean latency. section 12.2
Listing 12.6 Sampling SoC power during the soak
# 90 one-second samples of package power while the soak loop runs sudo powermetrics --samplers cpu_power,gpu_power -i 1000 -n 90
This command samples package power once per second during a sustained run; treat the result as an estimate rather than sensor-grade power. section 12.2
