liveBook · Manning

Chapter 12: Delivering Proven Results with a Four-Bit LLM Server and an On-Device Vision Pipeline

What you will learn in chapter 12 of Quantization and Fast Inference

The chapter introduces how to prove and monitor quantized deployments rather than relying on isolated speed benchmarks.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduces how to prove and monitor quantized deployments rather than relying on isolated speed benchmarks.

Key terms: forward pass, low-precision kernel, full-precision reference, offline gate, live observability, tokens per second, end-to-end path, latency target, hardware limits

12.1 Ship a four-bit large language model on a single GPU with tests

This section turns an AWQ four-bit checkpoint into a tested, hardware-compatible, monitored LLM endpoint and checks its performance and output quality.

Key terms: pre-ship gate, pytest, loopback, version-pinned, token-agreement function, AWQ, Marlin-family INT4 kernels, sm80, L4, T4

12.2 Ship a sub-second vision pipeline on device hardware

This section validates a sub-second INT8 vision pipeline on an Apple M3 by measuring user-visible latency, operation placement, sustained throughput, estimated power, and device telemetry.

Key terms: end-to-end p95, decode, pre-processing, Core ML forward pass, compute_units=ALL, MLComputePlan, preferred-device assignments, Apple Neural Engine (ANE), sustained loop, thermal retention

12.3 Compare tokens per second, memory, and quality across settings

This section sets up a controlled server format comparison and a separate server-versus-device comparison, while stating which formats were actually tested.

Key terms: FP16, AWQ four-bit, concurrency sweep, cross-plane view

12.3.1 Compare throughput and latency across formats

This section compares AWQ four-bit and FP16 serving on the same GPU, explaining their throughput, latency, and energy differences and the hardware condition for the four-bit gain.

Key terms: AWQ, FP16, concurrency, generation throughput, time-to-first-token p95, inter-token latency, memory-bandwidth-bound, weight traffic, batch, GPU power

12.3.2 Read the memory headroom, and the queue that never fills

This section explains how four-bit weights can increase serving headroom and why the measured 3B workload shows no queueing advantage over FP16.

Key terms: AWQ, FP16, KV cache, request queue, KV-cache occupancy, queue depth, headroom, knee, 3B

12.3.3 Weigh the quality cost

This subsection measures the quality cost of a faster four-bit AWQ endpoint so teams can weigh its performance gains against token-level drift.

Key terms: AWQ, canary, eight-prompt golden set, FP16 reference, token-level drift

12.3.4 Compare the device and the server without faking the axis

The section compares the L4 server and M3 device while showing why latency can follow a shared reporting discipline but their energy figures cannot share a meaningful axis.

Key terms: cross-plane comparison, energy per inference, sensor-grade, per-token generation, inference-only, end-to-end, latency target, forward pass

12.4 One last look back

The section closes the book by tying quantization choices to deployment tradeoffs and insisting that shipped results be measured and defensible.

Key terms: quantization, bits, speed, quality, drift, quantized, full precision, production, inference-only, conditions

12.5 Summary

Concepts explained in this chapter

← Chapter 11: Targeting Edge and Mobile Devices

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.