What you will learn in Quantization and Fast Inference
A practitioner’s guide to efficient AI
Chapter by chapter
Chapter 1: Facing the Efficiency Wall
This introduction explains why modern inference is constrained by data movement and why quantization is a practical response.
What you learn in chapter 1 →Chapter 2: Building Quantization from First Principles
This chapter introduction frames quantization as the translation from floating-point neural-network values to the small integers used for inference.
What you learn in chapter 2 →Chapter 3: Choosing What to Quantize and at What Granularity
This introduction explains why weights, activations, and the KV cache need different quantization choices and why scale granularity must balance accuracy with practical constraints.
What you learn in chapter 3 →Chapter 4: Applying Post-Training Quantization and Calibration
This introduction explains why post-training quantization depends on production-representative calibration data and why calibration can often avoid retraining.
What you learn in chapter 4 →Chapter 5: Recovering Accuracy with Quantization-Aware Training
The introduction explains when post-training quantization may need help and how quantization-aware training lets a model adapt to low precision.
What you learn in chapter 5 →Chapter 6: Porting Workflows Across Frameworks
This introduction explains why quantization intent can change across framework exports and why teams must verify that the resulting artifacts agree before deployment.
What you learn in chapter 6 →Chapter 7: Quantizing Large Language Models in Practice
The chapter introduction explains why LLM quantization must account for unusually large activation values and the memory demands of the KV cache.
What you learn in chapter 7 →Chapter 8: Pushing into Low Bits Safely
This introduction explains why some deployment constraints justify looking beyond established integer quantization methods and identifies the low-bit format families the chapter will examine.
What you learn in chapter 8 →Chapter 9: Shipping with the Right Toolchain
This introduction explains why a quantized model that works locally may still run at FP32 in production, making serving-time verification essential.
What you learn in chapter 9 →Chapter 10: Running on CPUs and Using GGUF
This introduction distinguishes CPU-first GGUF inference from graph-based deployment and explains why silent throughput losses matter in production.
What you learn in chapter 10 →Chapter 11: Targeting Edge and Mobile Devices
The introduction explains why edge deployment must account for sustained latency, energy use, and operator placement across a device’s compute blocks.
What you learn in chapter 11 →Chapter 12: Delivering Proven Results with a Four-Bit LLM Server and an On-Device Vision Pipeline
The chapter introduces how to prove and monitor quantized deployments rather than relying on isolated speed benchmarks.
What you learn in chapter 12 →Concepts explained in this book
- per-channel quantizationPer-channel quantization assigns separate quantization parameters to each tensor channel.
- quantization calibrationCalibration is the process of measuring representative model values to determine their actual numerical range for configuring quantization.
- block quantizationA technique that assigns scaling parameters separately to fixed-size blocks of numerical values rather than to an entire tensor.
- quantizationQuantization is the representation of numerical values such as neural-network weights and activations with reduced-precision formats.
- quantization scaleA scale factor is the multiplicative parameter that maps between quantized or fixed-point integers and their represented numerical values.
- quantization errorGranular error is the bounded quantization error caused by rounding a value to a nearby grid point.
- per-tensor quantizationPer-tensor quantization uses one set of quantization parameters for all values in a tensor.
- quantization-aware trainingQuantization-aware training incorporates simulated quantization effects while optimizing a model so it can retain accuracy at deployment precision.
- throughputThe amount of computation or output completed per unit of time, including its retention during sustained operation.
- latencyThe elapsed time required for a model or system to produce a result.
- affine quantizationAffine quantization maps real values to discrete integer levels using a scale and a zero-point offset.
- zero-point nudgeA zero-point nudge is the book's term for the adjustment of an asymmetric quantizer's zero-point to fit a shifted value distribution.
- zero-pointA zero-point is the integer level corresponding to real zero in an affine quantization mapping.
- dynamic quantizationDynamic range quantization stores model weights at reduced precision and handles activation ranges dynamically during inference.
- time to first tokenThe elapsed time from submitting a language-model request until its first generated token is available.
- outlier channelAn outlier channel or dimension is a feature coordinate whose activation magnitudes are substantially larger than those of most other coordinates.
- symmetric quantizationSymmetric quantization uses a zero-centered representation, typically forcing the weight zero-point to zero.
- asymmetric quantizationAsymmetric quantization maps values to a grid whose range and zero-point can be shifted to fit a non-centered distribution.
- floating-point valueA numerical representation that approximates real values using a significand and an exponent.
- hybrid consensusHybrid Consensus is a quantization strategy that applies symmetric quantization to weights and asymmetric quantization to activations.
- operator coverageThe proportion or extent of a model's operators affected by an optimization or quantization method.
- number roundingRounding maps a numerical value to a nearby representable value according to a specified tie-breaking or directional rule.
- memory bandwidthMemory bandwidth is the rate at which data can be transferred between memory and computational components.
- percentile-based calibrationPercentile calibration sets a quantization range using selected distribution percentiles, excluding values beyond those boundaries.
- AWQAWQ is a low-bit post-training quantization method that protects important weights or channels to preserve model quality.
- bit precisionThe number of bits used to represent each numerical value.
- quantization gridA quantization grid is a discrete set of representable numerical levels with a defined spacing.
- exact zero constraintThe exact zero constraint requires real zero to map to an integer value without quantization error.
- precisionThe numerical detail available to represent a value, often determined by the number of bits used.
- key-value cacheRuntime storage for attention keys and values from prior tokens, allowing an autoregressive transformer to reuse them when processing later tokens.
- per-token quantizationPer-token quantization assigns quantization parameters separately to the values associated with each input token.
- NormalFloat4NF4 is a 4-bit NormalFloat format that provides nonuniform representable levels suited to normally distributed neural-network weights.
- weight-only quantizationReducing the precision of model weights while leaving activations at higher precision.
- powerThe rate at which a device or system uses electrical energy.
- activation quantizationRepresenting neural-network activations with reduced-precision numerical formats.
- inferenceInference is the execution of a trained machine-learning model to produce outputs from inputs.
- activationA numerical output produced by a neural-network layer or operation and passed to subsequent computation.
- percentileA quantile is a value that divides a probability distribution so that a specified proportion of observations falls below it.
- float16A 16-bit floating-point format with 5 exponent bits and 10 fraction bits.
- GPTQGPTQ is a post-training quantization algorithm for compressing transformer model weights with controlled accuracy loss.
- energy per tokenEnergy per token is the energy consumed to generate one output token during model inference.
- FP8A family of 8-bit floating-point formats for reduced-precision numerical computation.
- bfloat16A 16-bit floating-point format with 8 exponent bits and 7 fraction bits, giving it a dynamic range similar to FP32.
- activation magnitudeActivation magnitude is the absolute size of a neural-network activation, considered per feature or channel.
- kv cache quantizationKV cache quantization is the reduced-precision representation of the attention keys and values stored for prior tokens.
- data driftDistribution mismatch is a difference between the value distribution represented by calibration data and that encountered in production.
- zero-point correctionA zero-point correction is the arithmetic adjustment that compensates for an activation zero-point in quantized matrix multiplication.
- fixed-point representationFixed-point representation encodes numerical values with a fixed scaling position between integer and fractional components.
- absmax quantizationA quantization range-estimation method that sets boundaries using the largest observed absolute value.
- decode phaseDecode is autoregressive inference that generates output tokens one at a time using the preceding context.
- error compensationError compensation redistributes quantization error from already processed values into remaining values to reduce reconstruction error.
- Hessian matrixA Hessian is a matrix of second-order partial derivatives describing the local curvature of a scalar-valued function.
- thermal kneeA thermal knee is the point during sustained computation at which thermal management causes a sharp reduction in performance.
- ONNX RuntimeONNX Runtime is an inference engine for executing models represented in the Open Neural Network Exchange format.
- q-formatQ-format notation specifies how a fixed-point number's bits are allocated between its integer and fractional components.
- right-shift operationA right shift moves an integer’s bits toward lower-order positions, approximately dividing by a power of two.
- rounding biasRounding bias is a half-step offset added before integer rescaling so that a shift approximates rounding rather than truncation.
- fake quantizationFakeQuantize is a module that applies simulated quantization and dequantization using configured quantization parameters during computation.
- calibration dataCalibration data is representative input data used to measure model values for configuring quantization.
- MSE-optimal range estimationA quantization range-estimation method that chooses boundaries to minimize mean squared error between original and reconstructed values.
- AMX-INT8Intel Advanced Matrix Extensions (AMX) are processor extensions that accelerate matrix operations using tile registers and dedicated instructions.
- AVX-512AVX-512 is a family of x86 instruction-set extensions providing 512-bit SIMD operations and related instruction subsets.
- concurrencyConcurrency is the number of requests or users being served simultaneously by a model instance.
- post-training quantizationPost-training quantization converts a trained model to lower-precision representations after training, typically using calibration data.
- ONNXONNX is an open standard for representing machine-learning models as portable computational graphs across frameworks and runtimes.
- ONNX dynamic quantizationquantize_dynamic is an ONNX Runtime API for dynamically quantizing an ONNX model.
- quantize_staticquantize_static is an ONNX Runtime API for statically quantizing an ONNX model.
- torch.onnx.exporttorch.onnx.export is a PyTorch API that exports a PyTorch model to the ONNX format.
- verification matrixA verification matrix is a structured assessment of model-artifact and runtime combinations, recording whether each combination loads and executes as expected.
- resolutionResolution is the spacing between adjacent representable numerical values.
- clipping-resolution trade-offThe clipping-resolution trade-off describes how narrowing a quantization range increases clipping while improving the spacing-based precision of values that remain in range.
- entropy calibrationEntropy calibration selects a quantization range by optimizing an entropy-based proxy for the distortion between original and quantized distributions.
- outlier ratioOutlier ratio is the ratio of the largest observed magnitude to a representative high percentile of value magnitudes, indicating the influence of the distribution tail.
- ternary modelA neural-network model whose weights use three discrete representable values.
- OpenVINO IROpenVINO IR is an intermediate model representation stored in XML and binary files for use with OpenVINO tools and runtimes.
- mixed precisionMixed precision uses different numerical precisions for different tensors or operations within a computation.
- deployment surfaceA deployment surface is the target device and operating environment on which a model is delivered and executed.
- SBC tierThe SBC tier is the deployment class based on single-board computers, with optional attached accelerators.
- small-box tierThe small-box tier is the deployment class for sustained workloads on fixed installations with discrete inference accelerators.
- marlin kernelA Marlin kernel is an optimized CUDA matrix-multiplication implementation for low-bit large-language-model inference.
- int4 quantizationA 4-bit integer representation used for low-precision numerical computation and storage.
- mathematical equivalenceEquivalence is the property of two models producing the same outputs for identical inputs.
- operator coverage analysisOperator coverage analysis identifies which operators in a model graph execute in a specified numerical format.
- clippingClipping limits a numerical value to a specified minimum and maximum so it remains within a representable range.
- data transferData movement is the transfer of data among storage, memory, and computational locations during computation.
- rangeRange is the interval of numerical values that a representation can encode.
- TFLiteConverterTFLiteConverter is the TensorFlow Lite API for converting models to TensorFlow Lite format.
- straight-through estimatorA straight-through estimator approximates gradients through a nondifferentiable operation using a surrogate derivative.
- IQ-quantizationIQ-quant is a family of llama.cpp importance-aware, low-bit quantization schemes that use non-uniform representations to compress model weights.
- integer computationArithmetic performed on integer-valued representations with discrete, fixed-step values.
- min-max calibrationMin-max calibration uses the minimum and maximum observed values to determine a quantization range and its parameters.
- QOperatorQOperator is an ONNX quantization format that replaces targeted operators with explicit integer operators.
- LLM.int8()LLM.int8() is a quantization method for large language models that uses 8-bit computation while handling activation outliers with higher precision.
- perturbation testPerturbation analysis evaluates how changing or disturbing one component affects a system's output or measured behavior.
- memory usageMemory footprint is the amount of memory occupied by a model or computation.
- energy per inferenceThe electrical energy consumed to complete one model inference, commonly measured in joules.
- AVX2AVX2 is an x86 SIMD instruction-set extension that adds 256-bit integer vector operations to AVX.
- delegateA delegate is a runtime component that executes supported model operations through a particular hardware backend or optimized operator library.
- activation-pattern diversityActivation-pattern diversity is the variety of numerical activation patterns elicited by different inputs to a model.
- activation rangeAn activation range is the interval of numerical values produced by a neural-network activation across inputs.
- calibration coverageA comparison between the activation range estimated during calibration and the corresponding range observed in production.
- runtime contractA runtime contract specifies the compatibility and execution expectations between a model artifact and the runtime that loads it.
- Triton model repositoryA Triton model repository is a directory structure containing model artifacts and configuration files that Triton uses to load and serve models.
- FP4A family of 4-bit floating-point formats for reduced-precision numerical representation.
- mean powerMean power is the average electrical power consumed by a device or workload over a measurement interval.
- TurboQuantTurboQuant is an online quantization method that compresses key-value caches using random rotations and a fixed codebook without calibration.
- preprocessingImage preprocessing is the transformation of an image before model execution, such as resizing or normalizing its pixel values.
- MLComputePlanMLComputePlan is a Core ML API for inspecting the compute plan and operation-level execution details of a model.
- E4M3An 8-bit floating-point format with one sign bit, four exponent bits, and three mantissa bits.
- saturationSaturation is the clamping of a value outside a numerical format's representable range to a boundary value.
- LiteRT-LMLiteRT-LM is a runtime for deploying and executing transformer models, with bundle-based inputs and streaming generation APIs.
- explicit quantizationExplicit quantization is a quantization mode in which quantization locations and parameters are represented in the model graph and honored by the inference-engine builder.
- implicit quantizationImplicit quantization is a quantization mode in which an inference-engine builder uses calibration information to choose which model layers execute at reduced precision.
- granularityThe level of tensor partitioning that determines which values share quantization parameters such as a scale factor.
- q4_K_M formatQ4_K_M is a llama.cpp quantization format that stores model weights primarily using four-bit K-quant blocks.
- IQ4_XSIQ4_XS is a llama.cpp importance-aware quantization format that represents model values using four-bit indices and a codebook.
- static quantizationStatic quantization determines quantization ranges during calibration before deployment and keeps the resulting parameters fixed at runtime.
- prefillPrefill is the initial computation that processes a language-model prompt before token generation begins.
- resource utilizationUtilization is the fraction of an instance's available capacity that is actively used over a period.
- 8-bit signed integerINT8 is an 8-bit signed or unsigned integer representation used for low-precision numerical computation and storage.
- NNCFNNCF is OpenVINO's neural network compression framework for optimizing models, including through quantization.
- OpenVINO CPU pluginThe OpenVINO CPU plugin is the OpenVINO backend that compiles and executes model graphs on CPUs.
- OpenVINO runtime model inspectorThe OpenVINO runtime model inspector examines a compiled runtime graph and reports the precision used by its operations.
- OpenVINO through ONNX RuntimeOpenVINO through ONNX Runtime is a deployment pattern that retains ONNX Runtime serving code and uses the OpenVINO execution provider.
- OpenVINOExecutionProviderThe OpenVINO execution provider is an ONNX Runtime backend that delegates model operations to OpenVINO.
- standalone OpenVINO deploymentStandalone OpenVINO deployment converts a model to IR, quantizes it with NNCF, and serves it through OpenVINO.
- step valueA step size is the spacing between adjacent representable levels in a uniform quantization grid.
- K-quantizationK-quant is a family of llama.cpp low-bit quantization schemes that store model values in structured blocks with associated scaling information.
- thread countThread count is the configured number of concurrent execution threads assigned to a computation or runtime.
- accuracy toleranceAccuracy tolerance is the amount of predictive accuracy degradation acceptable for a model or task.
- AVX-512 VNNIAVX-512 VNNI is an x86 instruction-set extension that adds low-precision integer dot-product operations to AVX-512.
- log-mel featuresA log-mel STFT is a short-time Fourier transform followed by conversion of spectral magnitudes to a logarithmic mel-frequency representation.
- context windowContext length is the number of input tokens a language model can process as context for an inference.
- memory transferMemory movement is the transfer of data between processors and levels of the memory hierarchy during computation.
- fixed-point arithmeticFixed-point arithmetic performs calculations on values encoded with a fixed scaling factor.
- double quantizationDouble quantization quantizes quantization parameters, such as block scales, to reduce their storage cost.
- low-rank adaptationLoRA is a parameter-efficient fine-tuning method that freezes pretrained weights and learns low-rank update matrices.
- QLoRAQLoRA is a parameter-efficient adaptation method that fine-tunes a quantized base model with low-rank adapters.
- full integer quantizationFull integer quantization represents model weights and activations using integer formats.
- representative datasetA representative dataset is a collection of sample inputs that reflects the data a model will process and supports quantization calibration.
- bits per weightBits per weight is the average number of bits used to represent each model weight in a quantized format.
- thermal throttlingSkin-temperature throttling reduces processor operating frequency or capacity when device-surface temperature reaches a thermal limit.
- llm_int8_thresholdllm_int8_threshold is a configuration parameter that sets the activation-magnitude threshold for identifying outlier dimensions in LLM.int8().
- validation protocolA validation protocol is a repeatable procedure for evaluating whether a model or transformation satisfies specified performance requirements.
- cross-runtime calibration driftCross-runtime calibration drift is a difference in quantization calibration results produced by distinct runtimes or compilers for the same model and input data.
- tier selectionTier selection is the choice of deployment class based on the constraints and capabilities of the target deployment surface.
- execution providerAn execution provider is an ONNX Runtime backend that assigns and executes model-graph operations on a particular hardware or software target.
- exponentThe scale-bearing component of a floating-point representation that determines the magnitude of its significand.
- mantissaThe part of a floating-point representation that stores the number's significant digits.
- equivalent transformationAn equivalent transformation changes internal model representations while preserving the model's output.
- sub-byte packingSub-byte packing combines multiple values narrower than one byte into byte-aligned storage units.
- superblockA superblock is a group of weights in a K-quant scheme that has an overall scale and contains smaller groups with their own scales.
- Apple MLXMLX is an array framework for machine-learning research and development, designed for efficient execution on Apple silicon.
- compute capabilityCompute capability is a GPU architecture identifier that specifies hardware features relevant to supported operations and kernels.
- Lloyd-Max algorithmThe Lloyd-Max algorithm iteratively assigns values to their nearest representative and updates representatives to minimize expected squared quantization error for a probability distribution.
- random rotationA random rotation is an orthogonal transformation selected randomly to change the coordinate orientation of numerical vectors.
- GGUFGGUF is a file format for packaging quantized machine-learning models and their associated metadata for inference runtimes.
- architectural sensitivityArchitectural sensitivity is the degree to which a model's output changes in response to perturbations of its parameters or activations.
- redundancyModel redundancy is the repeated encoding of information across a model's parameters, allowing some parameter perturbations without destroying the signal.
- signal-to-noise ratioSignal-to-noise ratio is the ratio of the magnitude or power of a desired signal to that of an accompanying noise component.
- phone tierThe phone tier is the deployment class comprising smartphones with integrated processing hardware and mobile-device constraints.
- QAT decision frameworkThe QAT decision framework is a diagnostic process for selecting an intervention based on quantization failure mode and bit-width.
- per-class accuracy analysisPer-class accuracy analysis evaluates a classifier's accuracy separately for each target class.
- gradientA gradient is a vector of derivatives describing how a function changes with respect to its inputs or parameters.
- progressive quantizationProgressive quantization introduces successively lower numerical precision over stages rather than applying the final precision immediately.
- normal distributionA normal distribution is a continuous probability distribution that is symmetric about its mean and has a bell-shaped density.
- dynamic rangeDynamic range is the span of magnitudes that a numerical representation can encode.
- simultaneous multithreadingSMT is a processor capability that allows multiple hardware threads to share the resources of one physical core.
- two's complementTwo's complement representation encodes signed integers so that the highest-order bit contributes a negative power-of-two weight.
- BitLinearBitLinear is the example's linear-layer implementation that quantizes weights and activations during training while preserving gradient flow.
- overload errorOverload error is quantization error caused when a value falls outside the representable range and is clipped to a boundary.
- imatrixAn imatrix is a collection of per-tensor activation-magnitude estimates used to weight quantization toward tensors important to model loss.
- palettizationWeight clustering groups neural-network weights around shared centroid values and represents the weights by cluster indices.
- AffineQuantizedTensorAffineQuantizedTensor is a TorchAO tensor subclass that represents quantized values together with their quantization parameters.
- compression ratioCompression ratio is the factor by which a model's stored representation is reduced relative to an original representation.
- quantize_()quantize_ is a TorchAO API that replaces eligible model weights with quantized tensor representations.
- runtime familyA runtime family is a class of model-execution software sharing a deployment backend or hardware-oriented execution model.
- throughput retentionSustained throughput retention is the fraction of initial throughput maintained during extended operation under thermal load.
- opsetAn opset is a versioned set of ONNX operators with defined semantics that ONNX runtimes implement.
- symbolic shape inferenceSymbolic shape inference determines tensor dimensions through a computational graph using symbolic expressions.
- q8_0Q8_0 is a block quantization format that stores 32 weights in 34 bytes using an FP16 scale and 32 signed INT8 values.
- clipped straight-through estimatorA clipped straight-through estimator passes gradients for values within a specified range and sets gradients outside that range to zero.
- perplexityPerplexity is a language-model metric measuring how well a probability model predicts a sequence, with lower values indicating better predictive performance.
- max absolute differenceThe largest absolute element-wise difference between two numerical outputs.
- mean absolute errorThe average absolute element-wise difference between two numerical outputs.
- QDQ formatQDQ format is an ONNX quantization representation that places QuantizeLinear and DequantizeLinear nodes around operators.
- top-1 prediction agreementTop-1 prediction agreement is the fraction of inputs for which two models select the same highest-scoring class.
- codebookA quantization codebook is a collection of representative values used to encode and reconstruct quantized inputs.
- preprocessing and postprocessingPre- and post-processing are the operations that transform model inputs before inference and model outputs afterward.
- ternary quantizationA technique that represents numerical values using three levels, commonly negative one, zero, and positive one.
- warmupLearning-rate warmup gradually increases the learning rate from a smaller initial value during an early phase of training.
- processor affinityCPU affinity is a policy or mechanism that restricts a process or thread to specified processor units.
- thread-count sweepA thread-count sweep measures a workload across multiple configured thread counts to identify a suitable setting.
- .litertlm bundleA LiteRT-LM bundle is a model artifact in the FlatBuffers family that packages a transformer model with runtime metadata.
- compute-unit hintA compute-unit hint is a Core ML setting that limits which compute units the runtime may consider without dictating operation placement.
- feedforward networkA feed-forward network is a neural-network component that transforms representations through successive learned layers without recurrent connections.
- Jaccard similarityJaccard similarity is the size of the intersection of two sets divided by the size of their union.
- tactic selectionTactic selection is the process of choosing among alternative implementations for neural-network operations on a target device.
- anomalyAn outlier is a value that differs substantially from most values in a distribution.
- int16x8A quantization configuration that uses 16-bit integer activations and 8-bit integer weights.
- CPUA CPU is a general-purpose processor that executes program instructions and arithmetic operations.
- batchingBatching processes multiple requests together so they can share model computation or data reads.
- BitNet b1.58 2B4TBitNet b1.58 2B4T is a 2-billion-parameter language model trained on 4 trillion tokens with ternary weights.
- cosine annealingCosine annealing is a schedule that smoothly decreases the learning rate along a cosine-shaped curve.
- conversion pipelineA conversion pipeline is a sequence of processing steps that transforms a model into a format or implementation suitable for a target runtime or accelerator.
- token-wise structureToken-wise variation is a pattern in which numerical magnitudes differ across token positions.
- batch normalization foldingBatch normalization folding is a graph transformation that incorporates batch-normalization parameters into an adjacent operation such as a convolution.
- quantized Johnson-Lindenstrauss correctionA quantized Johnson-Lindenstrauss correction uses quantized random projections to produce unbiased estimates of inner products.
- observerAn observer is a software component that records representative tensor statistics during calibration to determine quantization parameters.
- Q6_KQ6_K is a k-quant format that stores 256 weights per superblock using 6-bit codes and hierarchical scales.
- get_tensor_detailsA TensorFlow Lite interpreter method that returns metadata about the interpreter’s tensors.
- NVFP4NVFP4 is NVIDIA's 4-bit floating-point format that stores block scales in FP8 and a tensor-wide scale in FP32.
- residual connectionA residual connection adds a block's input to the output of a transformation applied to that input.
- state_dictstate_dict is a PyTorch interface for representing a module's parameters and persistent buffers as a mapping of names to tensors.
- model artifactA model artifact is a packaged, deployable representation of a machine-learning model and the information needed to execute it.
- latency budgetA latency budget is a maximum allowable elapsed time for a computation or service operation.
- range estimationA range estimation algorithm determines the numerical bounds used to configure a quantizer from observed model values.
- importance matrixNumerical data representing the relative importance of model weights for guiding quantization.
- TensorRT engineA TensorRT engine is a compiled model artifact optimized for execution by TensorRT on supported hardware.
- tokenizer configurationA tokenizer configuration specifies vocabulary, special-token IDs, and normalization rules for converting text into model tokens.
- powermetricspowermetrics is a macOS command-line utility that reports system and processor performance and power-related measurements.
- layer sensitivity analysisLayer sensitivity analysis measures how perturbing or quantizing individual layers affects model behavior or accuracy.
- affine transformationAn affine transformation is a mapping that combines a linear transformation with an additive offset.
- sub-scaleA sub-scale is a quantization scale applied to a smaller chunk of weights within a hierarchical quantization scheme.
- OpenVINOOpenVINO is an Intel toolkit for optimizing and deploying machine-learning models across diverse hardware targets.
- TensorRTTensorRT is an NVIDIA software development kit for optimizing and executing neural-network inference on GPUs.
- Core MLCore ML is Apple's framework for integrating machine-learning models into applications across its platforms.
- memory headroomMemory headroom is the unused memory capacity available before a system reaches its memory limit.
- L4 GPUL4 is an NVIDIA data-center GPU designed for accelerated inference and other parallel workloads.
- mean squared errorMean squared error is the average squared difference between corresponding numerical values.
- memory layoutMemory layout is the arrangement of data elements at addresses in memory.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
