liveBook · Manning

What you will learn in Quantization and Fast Inference

A practitioner’s guide to efficient AI

by Vivek Kalyanarangan · Manning Early Access Program (MEAP)

Chapter by chapter

Chapter 1: Facing the Efficiency Wall

This introduction explains why modern inference is constrained by data movement and why quantization is a practical response.

1.1 The cost crisis in memory, latency, and power · 1.2 Why quantization is the practical response · 1.3 Mapping Floating Point vs Integer · 1.3.1 Floating point's hidden costs · 1.3.2 What integers give up, and what they gain · 1.4 Summary

What you learn in chapter 1 →

Chapter 2: Building Quantization from First Principles

This chapter introduction frames quantization as the translation from floating-point neural-network values to the small integers used for inference.

2.1 The fixed-point world · 2.2 Define the affine mapping with scale and zero point · 2.3 Characterize granular and overload errors · 2.4 Compare symmetric and asymmetric choices · 2.5 End-to-end INT8 inference in PyTorch · 2.6 Summary

What you learn in chapter 2 →

Chapter 3: Choosing What to Quantize and at What Granularity

This introduction explains why weights, activations, and the KV cache need different quantization choices and why scale granularity must balance accuracy with practical constraints.

3.1 Identify where precision loss matters · 3.2 Quantize weights with per-tensor and per-channel choices · 3.3 Quantize activations under outliers and dynamic ranges · 3.4 Quantize transformer key-value cache with mixed precision · 3.5 Use group and block schemes and understand memory layout · 3.6 Follow a decision checklist for granularity · 3.7 Summary

What you learn in chapter 3 →

Chapter 4: Applying Post-Training Quantization and Calibration

This introduction explains why post-training quantization depends on production-representative calibration data and why calibration can often avoid retraining.

4.1 Know when post-training quantization is enough · 4.2 Build a calibration set that actually represents production · 4.3 Estimate ranges (absolute max, percentile, and MSE) · 4.4 Validate accuracy and size with a repeatable protocol · 4.5 Summary

What you learn in chapter 4 →

Chapter 5: Recovering Accuracy with Quantization-Aware Training

The introduction explains when post-training quantization may need help and how quantization-aware training lets a model adapt to low precision.

5.1 Recognize when post-training methods fall short · 5.1.1 The accuracy cliff · 5.1.2 Diagnostic signatures of PTQ failure · 5.2 Train with fake quantization and straight-through estimators · 5.2.1 Build a complete fake-quantized layer · 5.2.2 The QAT training loop · 5.3 Schedule training to minimize cost · 5.3.1 Warm up the learning rate · 5.3.2 Put the schedule together · 5.4 Use per-channel strategies for convolution and linear layers · 5.4.1 The gradient survival problem · 5.4.2 Measure the training impact · 5.5 Adapt transformers efficiently · 5.6 Summary

What you learn in chapter 5 →

Chapter 6: Porting Workflows Across Frameworks

This introduction explains why quantization intent can change across framework exports and why teams must verify that the resulting artifacts agree before deployment.

6.1 Know where frameworks diverge · 6.2 Run the PyTorch and TorchAO path end-to-end · 6.3 Run the TensorFlow path with the model optimization toolkit · 6.4 Export to ONNX and drive ONNX Runtime quantization · 6.5 Verify equivalence and diagnose coverage · 6.6 Summary

What you learn in chapter 6 →

Chapter 7: Quantizing Large Language Models in Practice

The chapter introduction explains why LLM quantization must account for unusually large activation values and the memory demands of the KV cache.

7.1 Handle outliers and key-value cache considerations · 7.2 Apply outlier-aware weight paths · 7.3 Apply Hessian-aware groupwise methods (GPTQ) · 7.4 Apply activation-aware weight protection (AWQ) · 7.5 Compress KV cache with vector quantization (TurboQuant) · 7.6 From method comparison to deployment choice · 7.7 Summary

What you learn in chapter 7 →

Chapter 8: Pushing into Low Bits Safely

This introduction explains why some deployment constraints justify looking beyond established integer quantization methods and identifies the low-bit format families the chapter will examine.

8.1 Decide when to go below eight bits · 8.2 Use FP8 formats and understand kernel caveats · 8.2.1 Value distribution: why non-uniform spacing matters · 8.2.2 Scaling is mandatory, and granularity matters · 8.2.3 Weight reconstruction: FP8 vs. INT8 · 8.2.4 FP8 deployment: 2× compression, zero quality loss · 8.3 Use FP4 with blockwise scaling without losing stability · 8.3.1 Seven values and a scale factor · 8.3.2 Blockwise scaling: from optional to mandatory · 8.3.3 Cross-format comparison: FP4 block-16 versus INT4 group-128 · 8.3.4 FP4 deployment: the quality cliff and how block scaling recovers · 8.4 Fine-tune with low-bit adapters · 8.5 Understand ternary quantization and where it fits · 8.5.1 Post-training ternary quantization fails · 8.5.2 BitLinear: making ternary trainable · 8.5.3 What you can run today · 8.6 Summary

What you learn in chapter 8 →

Chapter 9: Shipping with the Right Toolchain

This introduction explains why a quantized model that works locally may still run at FP32 in production, making serving-time verification essential.

9.1 Choose graphs, runtimes, and artifacts for deployment · 9.2 Use Optimum and ONNX Runtime quantizers the right way · 9.3 Build TensorRT engines with real integer execution · 9.4 Deploy on Intel CPU with OpenVINO and NNCF · 9.5 Package models for serving and distribution · 9.6 Summary

What you learn in chapter 9 →

Chapter 10: Running on CPUs and Using GGUF

This introduction distinguishes CPU-first GGUF inference from graph-based deployment and explains why silent throughput losses matter in production.

10.1 Know when CPU wins on cost and scale-out · 10.2 Understand the GGUF format and its variants · 10.2.1 Three quant families, three generations · 10.2.2 Four sections of the GGUF file · 10.2.3 Metadata makes the file self-describing · 10.2.4 The packing is not uniform · 10.2.5 Inside one block: from bytes to a weight · 10.3 Convert large language models to CPU-friendly artifacts · 10.3.1 The pipeline at a glance · 10.3.2 F16 export and the four quantization runs · 10.3.3 The imatrix is the IQ-quant prerequisite · 10.3.4 Verify size, perplexity, fluency, and decode throughput · 10.4 Compare kernel families and what they imply · 10.4.1 Three builds, one CPU · 10.4.2 Inside one Q4_K dot product · 10.4.3 Decode throughput across the matrix · 10.4.4 The IQ4_XS / Q4_K_M crossover · 10.4.5 What to ship · 10.5 Set realistic expectations for throughput and memory · 10.5.1 Runtime memory: weights, KV cache, and compute buffer · 10.5.2 Thread count and the cloud-CPU exception · 10.6 Summary

What you learn in chapter 10 →

Chapter 11: Targeting Edge and Mobile Devices

The introduction explains why edge deployment must account for sustained latency, energy use, and operator placement across a device’s compute blocks.

11.1 Set latency and power budgets · 11.1.1 The latency × power design space · 11.1.2 Sustained throughput and the thermal knee · 11.1.3 The trust ladder for power numbers · 11.1.4 The budget filter, applied · 11.2 Android with TFLite and LiteRT-LM · 11.2.1 Four paths, four graphs · 11.2.2 The Pixel runtime matrix · 11.2.3 LiteRT-LM as TFLite's transformer sibling · 11.3 Apple silicon with Core ML and MLX · 11.3.1 The compute-unit dispatch knob · 11.3.2 The ANE op-support cliff · 11.3.3 MLX as Core ML's transformer sibling · 11.3.4 Sustained throughput and the palettize cliff · 11.4 Phone, SBC, and small-box tiers · 11.5 Pre and post-processing · 11.6 Summary

What you learn in chapter 11 →

Chapter 12: Delivering Proven Results with a Four-Bit LLM Server and an On-Device Vision Pipeline

The chapter introduces how to prove and monitor quantized deployments rather than relying on isolated speed benchmarks.

12.1 Ship a four-bit large language model on a single GPU with tests · 12.2 Ship a sub-second vision pipeline on device hardware · 12.3 Compare tokens per second, memory, and quality across settings · 12.3.1 Compare throughput and latency across formats · 12.3.2 Read the memory headroom, and the queue that never fills · 12.3.3 Weigh the quality cost · 12.3.4 Compare the device and the server without faking the axis · 12.4 One last look back · 12.5 Summary

What you learn in chapter 12 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.