Chapter 12: Delivering Proven Results with a Four-Bit LLM Server and an On-Device Vision Pipeline
The chapter introduces how to prove and monitor quantized deployments rather than relying on isolated speed benchmarks.
After this chapter you will be able to
- Limits of isolated benchmarks. Explain why a fast forward-pass benchmark alone does not prove that a quantized model is ready to ship.
- Offline gates and live observability. Distinguish what an offline deployment gate checks from what live observability catches after release.
- User latency and hardware context. Distinguish inference-only performance numbers from user-facing latency, and explain why reported results need hardware limits alongside them.
- Pre-ship deployment gate. Explain what the GPU-free pytest gate checks before deployment and why those checks matter for trustworthy, safe serving.
- Four-bit serving hardware floor. Explain why the AWQ artifact serves with four-bit execution on the L4 but may not do so on a T4.
- Live serving telemetry. Trace how the running server and GPU provide measurements to the dashboard without adding instrumentation to the model server.
- Concurrency sweep interpretation. Describe what the rising-concurrency sweep measures and what its results do and do not show about this build.
- Greedy reference quality canary. Explain how the live canary compares the four-bit endpoint with an FP16 reference and why both runs use greedy decoding.
- End-to-End Latency Budget. Explain why the pipeline is judged by end-to-end p95 latency rather than Core ML inference time, and identify which stages dominate its measured latency.
- Compiled Operation Placement. Explain how the section checks where Core ML operations prefer to run and why its CPU-heavy operation count does not by itself establish an accelerator-routing failure.
- Sustained Throughput Retention. Describe how the section tested for thermal throttling and interpret why the measured retention exceeded 100% without generalizing it to heavier workloads.
- Comparison axes and evidence limits. Describe the two comparison axes and explain which format claims the captured runs can support.
- Matched Format Performance Comparison. Describe why the AWQ and FP16 results can be compared directly, and summarize their throughput and first-token latency differences.
- Inter-Token Latency and Bottlenecks. Explain how inter-token latency changes with concurrency for each format and why AWQ's throughput lead narrows as load rises.
- Energy Per Token Advantage. Explain how energy per token is calculated and why AWQ uses less energy per token despite drawing nearly the same GPU power as FP16.
- Weight Memory and Queueing. Explain how AWQ's smaller weights can delay request queueing by leaving more memory for vLLM's KV cache.
- Measured Headroom Without Saturation. Describe what the KV-cache and queue measurements show for the 3B workload, and explain why they do not reveal a four-bit advantage.
- Measured AWQ Quality Tradeoff. Explain how the canary measures the four-bit endpoint’s agreement with FP16 and how that result informs a deployment decision alongside AWQ’s speed, latency, and energy benefits.
- Cross-plane energy comparability. Explain why the section declines to plot server and device energy per inference on a shared axis.
- Inference and end-to-end timing. Distinguish what inference-only and end-to-end performance numbers are for in both builds.
- Quantization Deployment Tradeoffs. Explain why quantization helps models fit deployment constraints and why no single format is always best.
- Verified Quantized Execution. Explain why a deployed model must be checked for actual quantized execution rather than assumed to run that way.
- Defensible User-Facing Measurements. Describe how to report performance results that reflect what users experience and can be defended.
What it builds on
- Apply activation-aware weight protection (AWQ) (chapter 7) · The AWQ weight artifact used by this build.
- Validate accuracy and size with a repeatable protocol (chapter 4) · Validating quantized-model quality with a repeatable comparison.
- Pre and post-processing (chapter 11) · The preprocessing work surrounding on-device inference.
- The compute-unit dispatch knob (chapter 11) · Core ML compute-unit dispatch on Apple silicon.
- Sustained throughput and the thermal knee (chapter 11) · The distinction between short-run speed and sustained throughput at a thermal knee.
- The trust ladder for power numbers (chapter 11) · The need to distinguish levels of trust in power numbers.
- From method comparison to deployment choice (chapter 7) · Comparing quantization methods to make a deployment choice.
Section by section
Introduction
The chapter introduces how to prove and monitor quantized deployments rather than relying on isolated speed benchmarks.
- Limits of isolated benchmarks: Explain why a fast forward-pass benchmark alone does not prove that a quantized model is ready to ship.
Why it matters: It keeps a promising benchmark from being mistaken for proof of a working deployment. - Offline gates and live observability: Distinguish what an offline deployment gate checks from what live observability catches after release.
Why it matters: Tests and monitoring catch different failures, so neither replaces the other. - User latency and hardware context: Distinguish inference-only performance numbers from user-facing latency, and explain why reported results need hardware limits alongside them.
Why it matters: It helps readers judge whether a reported speed reflects user experience and the hardware on which it was measured.
Key terms: forward pass, low-precision kernel, full-precision reference, offline gate, live observability, tokens per second, end-to-end path, latency target, hardware limits
12.1 Ship a four-bit large language model on a single GPU with tests
This section turns an AWQ four-bit checkpoint into a tested, hardware-compatible, monitored LLM endpoint and checks its performance and output quality.
- Pre-ship deployment gate: Explain what the GPU-free pytest gate checks before deployment and why those checks matter for trustworthy, safe serving.
Why it matters: A fast endpoint is not ready to ship if its dashboard is misleading, its telemetry is exposed, or its reported results are untrustworthy. - Four-bit serving hardware floor: Explain why the AWQ artifact serves with four-bit execution on the L4 but may not do so on a T4.
Why it matters: A four-bit checkpoint alone does not guarantee four-bit execution or its memory-traffic benefit. - Live serving telemetry: Trace how the running server and GPU provide measurements to the dashboard without adding instrumentation to the model server.
Why it matters: These separate runtime and hardware measurements let operators watch the deployed endpoint rather than infer its behavior from the checkpoint. - Concurrency sweep interpretation: Describe what the rising-concurrency sweep measures and what its results do and do not show about this build.
Why it matters: The sweep supports a claim about this endpoint under rising load without mistaking unused memory for a tested capacity limit. - Greedy reference quality canary: Explain how the live canary compares the four-bit endpoint with an FP16 reference and why both runs use greedy decoding.
Why it matters: Speed dashboards cannot show whether a redeployed endpoint still produces outputs consistent with the reference.
Key terms: pre-ship gate, pytest, loopback, version-pinned, token-agreement function, AWQ, Marlin-family INT4 kernels, sm80, L4, T4
12.2 Ship a sub-second vision pipeline on device hardware
This section validates a sub-second INT8 vision pipeline on an Apple M3 by measuring user-visible latency, operation placement, sustained throughput, estimated power, and device telemetry.
- End-to-End Latency Budget: Explain why the pipeline is judged by end-to-end p95 latency rather than Core ML inference time, and identify which stages dominate its measured latency.
Why it matters: Measuring the whole path prevents a fast model call from hiding the stage that actually limits the user experience. - Compiled Operation Placement: Explain how the section checks where Core ML operations prefer to run and why its CPU-heavy operation count does not by itself establish an accelerator-routing failure.
Why it matters: Checking the compiled plan avoids assuming that requesting an accelerator proves it ran the model, while interpreting the counts avoids a false failure diagnosis. - Sustained Throughput Retention: Describe how the section tested for thermal throttling and interpret why the measured retention exceeded 100% without generalizing it to heavier workloads.
Why it matters: A short latency pass cannot establish that throughput will hold after a device warms up. - Estimated SoC Power: Explain what the reported power figure measures and why the section labels it estimated wherever it appears.
Why it matters: The estimate label keeps a useful power observation from being presented as a sensor-grade measurement. - Hand-Rolled Device Telemetry: Describe how the pipeline makes its validation measurements available to the observability stack despite Core ML and MLX lacking Prometheus-formatted exports.
Why it matters: The hand-built export path makes the device's performance and measurement qualifications visible after the run.
Key terms: end-to-end p95, decode, pre-processing, Core ML forward pass, compute_units=ALL, MLComputePlan, preferred-device assignments, Apple Neural Engine (ANE), sustained loop, thermal retention
12.3 Compare tokens per second, memory, and quality across settings
This section sets up a controlled server format comparison and a separate server-versus-device comparison, while stating which formats were actually tested.
- Comparison axes and evidence limits: Describe the two comparison axes and explain which format claims the captured runs can support.
Why it matters: Keeping the axes and unrun formats distinct prevents the results from supporting claims they cannot establish.
Key terms: FP16, AWQ four-bit, concurrency sweep, cross-plane view
12.3.1 Compare throughput and latency across formats
This section compares AWQ four-bit and FP16 serving on the same GPU, explaining their throughput, latency, and energy differences and the hardware condition for the four-bit gain.
- Matched Format Performance Comparison: Describe why the AWQ and FP16 results can be compared directly, and summarize their throughput and first-token latency differences.
Why it matters: A controlled comparison makes the performance difference meaningful rather than an artifact of different hardware or requests. - Inter-Token Latency and Bottlenecks: Explain how inter-token latency changes with concurrency for each format and why AWQ's throughput lead narrows as load rises.
Why it matters: The latency trend explains both the large four-bit advantage at light load and its smaller advantage at high load. - Energy Per Token Advantage: Explain how energy per token is calculated and why AWQ uses less energy per token despite drawing nearly the same GPU power as FP16.
Why it matters: Energy per token connects the throughput gain to the cost of serving tokens, even when the card's wattage does not fall. - Four-Bit Kernel Requirement: State the condition under which the measured AWQ serving advantage applies.
Why it matters: A four-bit artifact alone does not guarantee the measured performance on every GPU.
Key terms: AWQ, FP16, concurrency, generation throughput, time-to-first-token p95, inter-token latency, memory-bandwidth-bound, weight traffic, batch, GPU power
12.3.2 Read the memory headroom, and the queue that never fills
This section explains how four-bit weights can increase serving headroom and why the measured 3B workload shows no queueing advantage over FP16.
- Weight Memory and Queueing: Explain how AWQ's smaller weights can delay request queueing by leaving more memory for vLLM's KV cache.
Why it matters: It explains what serving benefit a smaller weight format could provide beyond reducing model size. - Measured Headroom Without Saturation: Describe what the KV-cache and queue measurements show for the 3B workload, and explain why they do not reveal a four-bit advantage.
Why it matters: It prevents an empty queue from being mistaken for evidence that four-bit compression has no serving value.
Key terms: AWQ, FP16, KV cache, request queue, KV-cache occupancy, queue depth, headroom, knee, 3B
12.3.3 Weigh the quality cost
This subsection measures the quality cost of a faster four-bit AWQ endpoint so teams can weigh its performance gains against token-level drift.
- Measured AWQ Quality Tradeoff: Explain how the canary measures the four-bit endpoint’s agreement with FP16 and how that result informs a deployment decision alongside AWQ’s speed, latency, and energy benefits.
Why it matters: A measured agreement result lets a team decide whether AWQ’s performance gains justify the observed drift.
Key terms: AWQ, canary, eight-prompt golden set, FP16 reference, token-level drift
12.3.4 Compare the device and the server without faking the axis
The section compares the L4 server and M3 device while showing why latency can follow a shared reporting discipline but their energy figures cannot share a meaningful axis.
- Cross-plane energy comparability: Explain why the section declines to plot server and device energy per inference on a shared axis.
Why it matters: A shared chart can imply that unlike measurements represent the same kind of cost. - Inference and end-to-end timing: Distinguish what inference-only and end-to-end performance numbers are for in both builds.
Why it matters: Using a fast model-only number as a user-facing latency promise hides the rest of the wait.
Key terms: cross-plane comparison, energy per inference, sensor-grade, per-token generation, inference-only, end-to-end, latency target, forward pass
12.4 One last look back
The section closes the book by tying quantization choices to deployment tradeoffs and insisting that shipped results be measured and defensible.
- Quantization Deployment Tradeoffs: Explain why quantization helps models fit deployment constraints and why no single format is always best.
Why it matters: A smaller model is useful only if its speed and quality suit the deployment. - Verified Quantized Execution: Explain why a deployed model must be checked for actual quantized execution rather than assumed to run that way.
Why it matters: A deployment claim is unreliable if the model did not execute in the claimed precision. - Defensible User-Facing Measurements: Describe how to report performance results that reflect what users experience and can be defended.
Why it matters: Others can rely on a performance claim only when it represents the user-facing path and its limits are clear.
Key terms: quantization, bits, speed, quality, drift, quantized, full precision, production, inference-only, conditions
12.5 Summary
Concepts explained in this chapter
- latencyThe elapsed time required for a model or system to produce a result.
- marlin kernelA Marlin kernel is an optimized CUDA matrix-multiplication implementation for low-bit large-language-model inference.
- MLComputePlanMLComputePlan is a Core ML API for inspecting the compute plan and operation-level execution details of a model.
- preprocessingImage preprocessing is the transformation of an image before model execution, such as resizing or normalizing its pixel values.
- throughputThe amount of computation or output completed per unit of time, including its retention during sustained operation.
- energy per tokenEnergy per token is the energy consumed to generate one output token during model inference.
- int4 quantizationA 4-bit integer representation used for low-precision numerical computation and storage.
- quantizationQuantization is the representation of numerical values such as neural-network weights and activations with reduced-precision formats.
- percentileA quantile is a value that divides a probability distribution so that a specified proportion of observations falls below it.
- bits per weightBits per weight is the average number of bits used to represent each model weight in a quantized format.
- inferenceInference is the execution of a trained machine-learning model to produce outputs from inputs.
- quantization calibrationCalibration is the process of measuring representative model values to determine their actual numerical range for configuring quantization.
- zero-pointA zero-point is the integer level corresponding to real zero in an affine quantization mapping.
- concurrencyConcurrency is the number of requests or users being served simultaneously by a model instance.
- weight-only quantizationReducing the precision of model weights while leaving activations at higher precision.
- energy per inferenceThe electrical energy consumed to complete one model inference, commonly measured in joules.
- key-value cacheRuntime storage for attention keys and values from prior tokens, allowing an autoregressive transformer to reuse them when processing later tokens.
- saturationSaturation is the clamping of a value outside a numerical format's representable range to a boundary value.
- AWQAWQ is a low-bit post-training quantization method that protects important weights or channels to preserve model quality.
- L4 GPUL4 is an NVIDIA data-center GPU designed for accelerated inference and other parallel workloads.
- float16A 16-bit floating-point format with 5 exponent bits and 10 fraction bits.
- compute capabilityCompute capability is a GPU architecture identifier that specifies hardware features relevant to supported operations and kernels.
- powerThe rate at which a device or system uses electrical energy.
- thermal kneeA thermal knee is the point during sustained computation at which thermal management causes a sharp reduction in performance.
- thermal throttlingSkin-temperature throttling reduces processor operating frequency or capacity when device-surface temperature reaches a thermal limit.
- codebookA quantization codebook is a collection of representative values used to encode and reconstruct quantized inputs.
- floating-point valueA numerical representation that approximates real values using a significand and an exponent.
- precisionThe numerical detail available to represent a value, often determined by the number of bits used.
- quantization scaleA scale factor is the multiplicative parameter that maps between quantized or fixed-point integers and their represented numerical values.
- ternary modelA neural-network model whose weights use three discrete representable values.
- memory headroomMemory headroom is the unused memory capacity available before a system reaches its memory limit.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
