liveBook · Manning

Ternary model

As explained in Quantization and Fast Inference

A neural-network model whose weights use three discrete representable values.

Ternary models store weights as {−1, 0, +1}, requiring an information-theoretic 1.58 bits per weight. Their deployment appeal is CPU inference without multiplications: the hot path uses addition and subtraction. The constraint is severe: ordinary post-training rounding can destroy model quality. section 8.1 section 8.5.3

The practical decision depends on how the model reaches ternary weights:

  • Train with the ternary constraint from the start, or use quantization-aware fine-tuning; naively rounding an existing continuous-weight model is not a viable path. section 8.1 section 8.5.3
  • BitLinear quantizes weights to {−1, 0, +1} on each forward pass and routes gradients through a straight-through estimator. section 8.5.3
  • Check that the intended CPU runtime and packed format are available; bitnet.cpp runs inference using addition and subtraction without a GPU. section 8.5.3

BitNet b1.58 2B4T

This natively trained 2-billion-parameter model uses 0.45 GB in packed form, versus 5.18 GB for OPT-2.7B. WikiText-2 perplexities are in a similar range (20.27 versus 19.22), though different tokenizers make decimal-level comparison unreliable. By contrast, ternary post-training quantization raises OPT-6.7B perplexity from 16 to over 13,000. section 8.5.3 section 8.5.3

Table 8.1 Sub-8-bit format families
Property
Low-bit floats (FP8/FP4)
Distribution-matched (NF4)
Ternary
Representable values
Non-uniform (exponent–mantissa)
Normal quantiles (codebook)
{−1, 0, +1}
Bits per weight
8 (FP8) or 4 (FP4)
4
1.58
Scaling required
Yes (mandatory, range-critical)
Yes (blockwise, per-64)
Per-tensor absmax
Hardware requirement
H100+ (FP8), Blackwell (FP4)
Any (software codebook)
CPU-optimized (bitnet.cpp)
Post-training applicable
Yes
Yes
No—catastrophic quality loss
Primary use case
Inference throughput, training
Fine-tuning on consumer GPUs
CPU-native LLM inference

Compare ternary weights with low-bit floats and NF4 by representation, storage, scaling, hardware, and post-training suitability. section 8.1

How Quantization and Fast Inference covers it

How deep it goes
Explains ternary representation, training constraints, and gradient routing, but does not provide a complete training procedure.
Examples
Compares BitNet b1.58 2B4T with OPT-2.7B on packed storage and WikiText-2 perplexity, and shows OPT-6.7B PTQ failure.
What you can do after
Evaluate whether ternary inference fits a deployment by checking training path, CPU runtime, packed size, and measured quality.
How it connects
Connects ternary weights to quantization-aware training, BitLinear, and the straight-through estimator.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.