liveBook · Manning

Energy per inference

As explained in Quantization and Fast Inference

The electrical energy consumed to complete one model inference, commonly measured in joules.

Energy per inference is the electrical energy used to complete one model inference, commonly expressed in joules. For edge deployments, it can be a more useful cost axis than mean power: the limiting budget depends on the workload. section 11.6

Power and energy constrain different workloads. Whisper-tiny on the M3 Neural Engine draws about 5.7 W, yet takes 30 times longer per inference than EfficientNet; for sequential ASR, energy per inference can become the binding constraint. Apply latency and power budgets together, then identify which axis actually limits the workload. section 11.1 section 11.6

Keep comparisons on a fair axis

On the same L4 server, AWQ INT4 roughly doubles peak throughput over FP16 and reduces energy from 45.4 to 24.5 mJ per token. Do not directly compare that server figure with the device estimate of 16.8 mJ per inference: the units describe different workloads, and one value is sensor-grade while the other is estimated. section 12.3.4 section 12.3.4

How Quantization and Fast Inference covers it

How deep it goes
Defines cost in joules per inference and contrasts it with mean power, but does not give a general measurement procedure.
Examples
Uses ASR timing and AWQ token-energy figures, and rejects a device/server comparison with mismatched units and telemetry.
What you can do after
Evaluate whether energy is the binding budget and whether compared figures share task units and measurement quality.
How it connects
Connects energy per inference to mean power, latency budgets, sustained-load thermal throttling, and workload type.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.