liveBook · Manning

Chapter 6: Porting Workflows Across Frameworks

What you will learn in chapter 6 of Quantization and Fast Inference

This introduction explains why quantization intent can change across framework exports and why teams must verify that the resulting artifacts agree before deployment.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains why quantization intent can change across framework exports and why teams must verify that the resulting artifacts agree before deployment.

Key terms: quantization workflows, default choices, accuracy, PyTorch, ONNX artifact, TensorFlow Lite, export targets, verify

6.1 Know where frameworks diverge

This section explains why quantized models can produce different results across frameworks, what practical parity means, and why teams must cross framework boundaries.

Key terms: INT8 quantization, per-tensor, per-axis (per-channel), symmetric, asymmetric, quantization grids, operator fusion, graph rewriting, Conv-BN-ReLU, batch norm folding

6.2 Run the PyTorch and TorchAO path end-to-end

This section uses TorchAO to create and evaluate INT8 PyTorch models, showing when its compression is useful, why eager-mode inference is not faster, and how to save the result.

Key terms: quantize_(), nn.Linear, AffineQuantizedTensor, INT8, per-channel scales, Conv2d, compression ceiling, per-channel FP32 scales, W8A8 dynamic, activation quantization

6.3 Run the TensorFlow path with the model optimization toolkit

This section runs TensorFlow Lite post-training quantization on MobileNetV2 and BERT, then uses the results to choose quantization modes and explain differences from TorchAO.

Key terms: TFLiteConverter, representative dataset, full integer quantization, PT2E, MobileNetV2, dynamic range, full integer, FP16, ReLU6, BERT-base

6.4 Export to ONNX and drive ONNX Runtime quantization

This section exports PyTorch models to ONNX, quantizes them with ONNX Runtime, and uses ResNet-18 and BERT results to show when static, dynamic, or selective quantization makes sense.

Key terms: ONNX, opset, .onnx file, ONNX Runtime, torch.onnx.export, dynamo=False, dynamo=True, TorchScript, torch.export, onnx.checker.check_model

6.5 Verify equivalence and diagnose coverage

This section verifies that quantized models ported across frameworks produce comparable outputs and diagnoses whether their operators actually run in INT8.

Key terms: equivalence, raw output tensors, logits, PyTorch FP32 baseline, max absolute difference, mean absolute difference, MSE, cosine similarity, top-1 prediction agreement, ONNX FP32

6.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 5: Recovering Accuracy with Quantization-Aware Training Chapter 7: Quantizing Large Language Models in Practice →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.