Ternary models store weights as {−1, 0, +1}, requiring an information-theoretic 1.58 bits per weight. Their deployment appeal is CPU inference without multiplications: the hot path uses addition and subtraction. The constraint is severe: ordinary post-training rounding can destroy model quality. section 8.1 section 8.5.3
The practical decision depends on how the model reaches ternary weights:
- Train with the ternary constraint from the start, or use quantization-aware fine-tuning; naively rounding an existing continuous-weight model is not a viable path. section 8.1 section 8.5.3
- BitLinear quantizes weights to {−1, 0, +1} on each forward pass and routes gradients through a straight-through estimator. section 8.5.3
- Check that the intended CPU runtime and packed format are available; bitnet.cpp runs inference using addition and subtraction without a GPU. section 8.5.3
BitNet b1.58 2B4T
This natively trained 2-billion-parameter model uses 0.45 GB in packed form, versus 5.18 GB for OPT-2.7B. WikiText-2 perplexities are in a similar range (20.27 versus 19.22), though different tokenizers make decimal-level comparison unreliable. By contrast, ternary post-training quantization raises OPT-6.7B perplexity from 16 to over 13,000. section 8.5.3 section 8.5.3
Table 8.1 Sub-8-bit format families
|
Property
|
Low-bit floats (FP8/FP4)
|
Distribution-matched (NF4)
|
Ternary
|
|
Representable values
|
Non-uniform (exponent–mantissa)
|
Normal quantiles (codebook)
|
{−1,
0, +1}
|
|
Bits per weight
|
8 (FP8) or 4 (FP4)
|
4
|
1.58
|
|
Scaling required
|
Yes (mandatory, range-critical)
|
Yes (blockwise, per-64)
|
Per-tensor absmax
|
|
Hardware requirement
|
H100+ (FP8), Blackwell (FP4)
|
Any (software codebook)
|
CPU-optimized (bitnet.cpp)
|
|
Post-training applicable
|
Yes
|
Yes
|
No—catastrophic quality loss
|
|
Primary use case
|
Inference throughput, training
|
Fine-tuning on consumer GPUs
|
CPU-native LLM inference
|
Compare ternary weights with low-bit floats and NF4 by representation, storage, scaling, hardware, and post-training suitability. section 8.1
