How deeply does Transformers in Action cover post-training quantization?

Compares quantization variants and trade-offs, but does not detail the quantization algorithm’s internal calculations.

Post-training quantization

Posttraining quantization converts a pretrained model to lower-precision representations without additional model training.

Post-training quantization converts a pretrained model to lower-precision representations without additional training, reducing inference memory and supporting resource-constrained or rapid deployment scenarios. section 8.3.3

Quantization choices involve distinct trade-offs:

Nonuniform quantization uses variable-sized intervals adapted to weight or layer distributions, potentially reducing quantization error. section 8.3.3
Quantization-aware training retrains with quantized weights and activations, preserving performance at lower precision but increasing computational cost. section 8.3.3
Post-training quantization avoids retraining and is less computationally demanding, but typically remains at 8-bit or 6-bit precision. section 8.3.3

Loading a 4-bit model

A bitsandbytes configuration loads Falcon 7B with 4-bit weights, selects FP4 quantization, uses bfloat16 computation, and records GPU memory before and after loading. section 8.3.3

Table 8.1 Comparison of data types

Data type	Bit width	Range	Memory reduction from FP32
float32	32 bits	≈ ± 3.4 × 10^{38}	–
float16	16 bits	≈ ± 6.5 × 10^{4}	50%
int8	8 bits	-128 to 127	75%
int4	4 bits	-8 to 7	87.5%

Compare float32, float16, int8, and int4 bit widths, numeric ranges, and stated memory reductions. section 8.3.3

How Transformers in Action covers it

liveBook manning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Domain-Specific Small Language ModelsA quantization method applied after training to reduce model numeric precision, typically to improve deployment efficiency.

Compare how 2 books explain post-training quantization →

Related concepts

GPTQincludes · GPTQ is a one-shot post-training weight-quantization method that uses calibration data and approximate second-order information to reduce precision while limiting accuracy loss.
quantizationkind of · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
quantization-aware trainingalternative to · Quantization-aware training optimizes a model while simulating or incorporating reduced-precision effects so it adapts to quantized representations.
calibration dataconfigured by · Calibration data is data used to measure, estimate, or adjust a model’s parameters, outputs, or behavior for a specified calibration objective.
SmoothQuantimplemented by · A post-training quantization technique that reduces memory footprint and speeds up inference by smoothing activation and weight ranges.
8-bit signed integeruses · An 8-bit signed integer is a fixed-width integer type representing whole-number values from −128 through 127.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.