Chapter 6: Porting Workflows Across Frameworks
This introduction explains why quantization intent can change across framework exports and why teams must verify that the resulting artifacts agree before deployment.
After this chapter you will be able to
- Cross-framework quantization drift. Explain how a model can lose accuracy when moved between frameworks even if each tool and pipeline stage appears to work correctly.
- Deliberate artifact verification. Justify producing artifacts deliberately for each required framework and checking that they agree before production, rather than relying on a single-framework workflow.
- Framework quantization defaults. Distinguish the default weight and activation quantization choices described for PyTorch eager mode, TensorFlow quantization-aware training, and ONNX Runtime static quantization. Explain why specifying INT8 does not guarantee the same quantization grid.
- Fusion and graph differences. Explain how frameworks' different operator-fusion and graph-rewriting rules can change quantized outputs, using the Conv-BN-ReLU example.
- Calibration range differences. Explain why implementations of the same named calibration method can produce different ranges and affect a network's outputs.
- Practical cross-framework parity. Describe what counts as parity between quantized artifacts and why it does not require bitwise-identical outputs.
- TorchAO weight conversion. Explain what TorchAO’s quantize_() changes in weight-only quantization and what it leaves unchanged.
- Architecture-dependent compression ceiling. Apply the section’s compression estimate using the fraction of parameters in nn.Linear, and explain why it favors BERT and TinyLlama over ResNet-18.
- Compression versus eager latency. Explain why smaller TorchAO models were slower in eager-mode PyTorch and what kind of execution is needed for a latency improvement.
- Converter and calibration workflow. Describe how TFLiteConverter produces quantized artifacts and how full integer calibration differs from PyTorch PT2E calibration.
- MobileNetV2 quantization tradeoffs. Explain why MobileNetV2 is a strong TFLite quantization target and compare its dynamic range, full integer, and FP16 results.
- Transformer activation range choice. Justify dynamic range rather than full integer quantization for BERT in this TFLite experiment.
- Cross-framework parity interpretation. Explain what the same-checkpoint BERT comparison establishes and why its artifact sizes differ.
- ONNX portability and runtimes. Explain how ONNX lets a model trained in one framework run through another deployment stack, and distinguish the ONNX format from ONNX Runtime.
- PyTorch export and validation. Describe how PyTorch exports a model to ONNX, why this section uses the legacy exporter, and how to check that an export still behaves like the original model.
- Static quantization for ResNet. Explain why ORT static quantization compresses ResNet-18 substantially more than dynamic quantization, and describe the static quantization pipeline used here.
- Dynamic quantization for BERT. Justify the dynamic quantization choice for BERT and trace why this section's generic static-quantization attempt failed.
- Output Equivalence Testing. Explain why matching accuracy is not enough to establish equivalence, and describe how to compare raw logits from identical inputs against a PyTorch FP32 baseline.
- Logit Divergence Measures. Describe what the five reported metrics reveal about output differences, including why a large absolute logit shift need not change a prediction.
- Export and Quantization Divergence. Explain how the unquantized ONNX comparison isolates export error, and contrast what the ResNet-18 and BERT-base results show about logit drift and prediction changes.
- Operator Coverage Interpretation. Describe how ONNX graph inspection identifies quantized operators in QDQ and operator-replacement formats, and explain why operator-count coverage alone does not establish likely speedup.
What it builds on
- Quantize weights with per-tensor and per-channel choices (chapter 3) · Assumes the distinction between per-tensor and per-channel weight quantization.
- Compare symmetric and asymmetric choices (chapter 2) · Assumes the distinction between symmetric and asymmetric quantization.
- Estimate ranges (absolute max, percentile, and MSE) (chapter 4) · Assumes familiarity with estimating quantization ranges.
- Quantize activations under outliers and dynamic ranges (chapter 3) · The distinction between quantizing activations and quantizing weights.
- Build a calibration set that actually represents production (chapter 4) · Introduces representative calibration data.
- Validate accuracy and size with a repeatable protocol (chapter 4) · Repeatable accuracy and size comparisons are needed to judge a precision allocation.
Section by section
Introduction
This introduction explains why quantization intent can change across framework exports and why teams must verify that the resulting artifacts agree before deployment.
- Cross-framework quantization drift: Explain how a model can lose accuracy when moved between frameworks even if each tool and pipeline stage appears to work correctly.
Why it matters: A small, hard-to-trace accuracy loss can affect many production predictions. - Deliberate artifact verification: Justify producing artifacts deliberately for each required framework and checking that they agree before production, rather than relying on a single-framework workflow.
Why it matters: Checking agreement across the artifacts helps prevent export-time differences from reaching users.
Key terms: quantization workflows, default choices, accuracy, PyTorch, ONNX artifact, TensorFlow Lite, export targets, verify
6.1 Know where frameworks diverge
This section explains why quantized models can produce different results across frameworks, what practical parity means, and why teams must cross framework boundaries.
- Framework quantization defaults: Distinguish the default weight and activation quantization choices described for PyTorch eager mode, TensorFlow quantization-aware training, and ONNX Runtime static quantization. Explain why specifying INT8 does not guarantee the same quantization grid.
Why it matters: The same INT8 label can hide different numerical choices in your training and serving stacks. - Fusion and graph differences: Explain how frameworks' different operator-fusion and graph-rewriting rules can change quantized outputs, using the Conv-BN-ReLU example.
Why it matters: A changed prediction can arise from how a framework rewrites operations, not necessarily from a faulty export. - Calibration range differences: Explain why implementations of the same named calibration method can produce different ranges and affect a network's outputs.
Why it matters: Matching a calibration method's name is not enough to ensure matching quantized values. - Practical cross-framework parity: Describe what counts as parity between quantized artifacts and why it does not require bitwise-identical outputs.
Why it matters: It gives teams a testable standard for agreement without demanding impossible bit-for-bit equality. - Reasons for framework crossings: Explain why teams may need to move quantized models between frameworks despite the risk of numerical divergence.
Why it matters: Avoiding framework boundaries altogether is often not a practical deployment option.
Key terms: INT8 quantization, per-tensor, per-axis (per-channel), symmetric, asymmetric, quantization grids, operator fusion, graph rewriting, Conv-BN-ReLU, batch norm folding
6.2 Run the PyTorch and TorchAO path end-to-end
This section uses TorchAO to create and evaluate INT8 PyTorch models, showing when its compression is useful, why eager-mode inference is not faster, and how to save the result.
- TorchAO weight conversion: Explain what TorchAO’s quantize_() changes in weight-only quantization and what it leaves unchanged.
Why it matters: It clarifies what the one-line API actually produces before you compare its size, accuracy, or speed. - Architecture-dependent compression ceiling: Apply the section’s compression estimate using the fraction of parameters in nn.Linear, and explain why it favors BERT and TinyLlama over ResNet-18.
Why it matters: Counting where parameters live lets you estimate whether TorchAO will save meaningful space before running it. - Weight and activation tradeoffs: Distinguish the storage and accuracy observations for INT8 weight-only and dynamic W8A8 configurations on BERT.
Why it matters: It prevents confusing activation quantization with additional checkpoint compression. - Compression versus eager latency: Explain why smaller TorchAO models were slower in eager-mode PyTorch and what kind of execution is needed for a latency improvement.
Why it matters: A compressed checkpoint is not evidence that an eager-mode inference workload will run faster. - Portable quantized model saving: Describe how to save and reload a TorchAO model and why the section favors that route over full-model pickling.
Why it matters: A compressed model is useful as an artifact only if it can be reliably reloaded.
Key terms: quantize_(), nn.Linear, AffineQuantizedTensor, INT8, per-channel scales, Conv2d, compression ceiling, per-channel FP32 scales, W8A8 dynamic, activation quantization
6.3 Run the TensorFlow path with the model optimization toolkit
This section runs TensorFlow Lite post-training quantization on MobileNetV2 and BERT, then uses the results to choose quantization modes and explain differences from TorchAO.
- Converter and calibration workflow: Describe how TFLiteConverter produces quantized artifacts and how full integer calibration differs from PyTorch PT2E calibration.
Why it matters: The workflow determines both how little code you need and how much control you give up. - MobileNetV2 quantization tradeoffs: Explain why MobileNetV2 is a strong TFLite quantization target and compare its dynamic range, full integer, and FP16 results.
Why it matters: A mode that quantizes more tensors need not produce the smallest artifact. - Transformer activation range choice: Justify dynamic range rather than full integer quantization for BERT in this TFLite experiment.
Why it matters: The activation-range choice can determine whether a compressed transformer remains usable. - Cross-framework parity interpretation: Explain what the same-checkpoint BERT comparison establishes and why its artifact sizes differ.
Why it matters: Comparing ratios without checking artifact format and layer coverage can misattribute a difference to the framework itself. - Concrete tracing shape constraint: Describe the input-shape requirement when converting the TensorFlow BERT model and its serving implication.
Why it matters: A working conversion path can still impose input-shape constraints that matter in serving.
Key terms: TFLiteConverter, representative dataset, full integer quantization, PT2E, MobileNetV2, dynamic range, full integer, FP16, ReLU6, BERT-base
6.4 Export to ONNX and drive ONNX Runtime quantization
This section exports PyTorch models to ONNX, quantizes them with ONNX Runtime, and uses ResNet-18 and BERT results to show when static, dynamic, or selective quantization makes sense.
- ONNX portability and runtimes: Explain how ONNX lets a model trained in one framework run through another deployment stack, and distinguish the ONNX format from ONNX Runtime.
Why it matters: The distinction explains why exporting a model and choosing a runtime are separate decisions. - PyTorch export and validation: Describe how PyTorch exports a model to ONNX, why this section uses the legacy exporter, and how to check that an export still behaves like the original model.
Why it matters: A structurally accepted ONNX graph can still be unusable, so export needs an inference-level check. - Static quantization for ResNet: Explain why ORT static quantization compresses ResNet-18 substantially more than dynamic quantization, and describe the static quantization pipeline used here.
Why it matters: The operators a quantizer reaches determine whether a CNN gets meaningful compression. - Dynamic quantization for BERT: Justify the dynamic quantization choice for BERT and trace why this section's generic static-quantization attempt failed.
Why it matters: The CNN result does not generalize to transformers: operator coverage and changing activations favor a different quantization path. - Selective FP32 layer protection: Describe how ORT can leave selected nodes in FP32 and explain what the ResNet-18 precision-allocation experiments imply about when to use that option.
Why it matters: Selective FP32 protection is useful only when measured accuracy gains justify its model-size cost.
Key terms: ONNX, opset, .onnx file, ONNX Runtime, torch.onnx.export, dynamo=False, dynamo=True, TorchScript, torch.export, onnx.checker.check_model
6.5 Verify equivalence and diagnose coverage
This section verifies that quantized models ported across frameworks produce comparable outputs and diagnoses whether their operators actually run in INT8.
- Output Equivalence Testing: Explain why matching accuracy is not enough to establish equivalence, and describe how to compare raw logits from identical inputs against a PyTorch FP32 baseline.
Why it matters: Sample-level disagreement can affect downstream logic and regression tests even when reported accuracy is unchanged. - Logit Divergence Measures: Describe what the five reported metrics reveal about output differences, including why a large absolute logit shift need not change a prediction.
Why it matters: Using several metrics prevents a large score difference from being mistaken automatically for a changed decision. - Export and Quantization Divergence: Explain how the unquantized ONNX comparison isolates export error, and contrast what the ResNet-18 and BERT-base results show about logit drift and prediction changes.
Why it matters: A clean FP32 export baseline helps locate the source of divergence, while prediction agreement shows whether quantization drift matters for these models. - Operator Coverage Interpretation: Describe how ONNX graph inspection identifies quantized operators in QDQ and operator-replacement formats, and explain why operator-count coverage alone does not establish likely speedup.
Why it matters: Coverage weighted by computational cost is a better speedup diagnostic than an INT8 label or node count. - Coverage Patterns and Failure Diagnosis: Use the section's ResNet-18 and BERT-base coverage results to explain their different compression patterns, and describe how to triage export divergence, accuracy loss, missing speedup, or an unsupported quantized kernel.
Why it matters: The same quantization method can have very different effects across architectures, so the observed symptom should direct the next check.
Key terms: equivalence, raw output tensors, logits, PyTorch FP32 baseline, max absolute difference, mean absolute difference, MSE, cosine similarity, top-1 prediction agreement, ONNX FP32
6.6 Summary
Where this is used later
- Choose graphs, runtimes, and artifacts for deployment (chapter 9) · Choosing a deployment runtime capable of exploiting quantized execution.
- Android with TFLite and LiteRT-LM (chapter 11) · Informs the choice of a quantized mobile vision artifact.
- Handle outliers and key-value cache considerations (chapter 7) · Provides an example of why transformer activations need special attention.
- Use Optimum and ONNX Runtime quantizers the right way (chapter 9) · The manual ONNX export and static-quantization workflow that Optimum simplifies.
Concepts explained in this chapter
- operator coverageThe proportion or extent of a model's operators affected by an optimization or quantization method.
- ONNXONNX is an open standard for representing machine-learning models as portable computational graphs across frameworks and runtimes.
- ONNX RuntimeONNX Runtime is an inference engine for executing models represented in the Open Neural Network Exchange format.
- ONNX dynamic quantizationquantize_dynamic is an ONNX Runtime API for dynamically quantizing an ONNX model.
- quantize_staticquantize_static is an ONNX Runtime API for statically quantizing an ONNX model.
- torch.onnx.exporttorch.onnx.export is a PyTorch API that exports a PyTorch model to the ONNX format.
- mathematical equivalenceEquivalence is the property of two models producing the same outputs for identical inputs.
- operator coverage analysisOperator coverage analysis identifies which operators in a model graph execute in a specified numerical format.
- TFLiteConverterTFLiteConverter is the TensorFlow Lite API for converting models to TensorFlow Lite format.
- dynamic quantizationDynamic range quantization stores model weights at reduced precision and handles activation ranges dynamically during inference.
- full integer quantizationFull integer quantization represents model weights and activations using integer formats.
- representative datasetA representative dataset is a collection of sample inputs that reflects the data a model will process and supports quantization calibration.
- AffineQuantizedTensorAffineQuantizedTensor is a TorchAO tensor subclass that represents quantized values together with their quantization parameters.
- compression ratioCompression ratio is the factor by which a model's stored representation is reduced relative to an original representation.
- quantize_()quantize_ is a TorchAO API that replaces eligible model weights with quantized tensor representations.
- opsetAn opset is a versioned set of ONNX operators with defined semantics that ONNX runtimes implement.
- symbolic shape inferenceSymbolic shape inference determines tensor dimensions through a computational graph using symbolic expressions.
- QDQ formatQDQ format is an ONNX quantization representation that places QuantizeLinear and DequantizeLinear nodes around operators.
- max absolute differenceThe largest absolute element-wise difference between two numerical outputs.
- mean absolute errorThe average absolute element-wise difference between two numerical outputs.
- top-1 prediction agreementTop-1 prediction agreement is the fraction of inputs for which two models select the same highest-scoring class.
- state_dictstate_dict is a PyTorch interface for representing a module's parameters and persistent buffers as a mapping of names to tensors.
- weight-only quantizationReducing the precision of model weights while leaving activations at higher precision.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
