10 Running on CPUs and Using GGUF
This chapter covers
- Deciding when CPU inference wins on cost
- Reading the GGUF format and its quantization families
- Converting Hugging Face checkpoints to quantized GGUF
- Comparing AVX-2, AVX-512, and AMX kernels
- Diagnosing throughput loss from build flags and threads
We just worked through server-class deployment on x86 and NVIDIA silicon—ONNX Runtime, TensorRT, and OpenVINO—and deferred one large regime to this chapter: CPU-first LLM inference through llama.cpp and the GGUF file format.
That deferral was deliberate.
GGUF does not round-trip through ONNX. Its K-quant and IQ-quant weight schemes do not map onto operator-level INT8 dispatch. The runtime that consumes a GGUF file is a single-binary C++ inference engine, not a graph executor with execution providers. The file format, the runtime, and the operational surface are all different, but the end goal is getting low-precision compute to actually execute on the target hardware.
The following lesson is for the platform team shipping LLM inference without a GPU, for cost, for fleet bin-packing, or because the deployment target has no GPU at all. The failure mode to FP32 fallback is throughput left on the table. A wrong quantization choice, an AMX build flag for batch-1 decode, or a thread count copied from a bare-metal recommendation onto a cloud VM each costs 10–30% of what the silicon can deliver, with no error in any log.