GGUF
GGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.
LLMs in Production
GGUF is a model-file format designed to store quantized language models for efficient local loading and inference.
- How it is used
- Enables local, CPU-oriented model loading and inference on edge devices, including Raspberry Pi systems and phones.
- How deep it goes
- Explains conversion, tokenizer packaging, file management, and q4_K_M quantization, but defers detailed quantized-format coverage.
- Examples
- Uses a converted LLaVA model, q4_K_M quantization, and a Raspberry Pi network service as practical examples.
- What you can do after
- Build it from scratch by converting and quantizing a model for local service through llama.cpp.
Build Applications with Local AI Models on a Mac
A file format for storing machine-learning model tensors and metadata for inference, including quantized models.
- How it is used
- Packages fine-tuned models for local inference in Ollama.
- How deep it goes
- Explains quantization tradeoffs and outlines export and deployment without detailing GGUF file internals.
- Examples
- Compares quantization levels and illustrates a 7B model’s approximate memory requirements at different precisions.
- What you can do after
- Apply it via Unsloth and Ollama to export, create, and run a custom model locally.
Read how Build Applications with Local AI Models on a Mac explains GGUF →
Quantization and Fast Inference
GGUF is a file format for packaging quantized machine-learning models and their associated metadata for inference runtimes.
- How it is used
- Packages quantized models for CPU-first inference with llama.cpp, including deployments where GPU hardware is unavailable.
- How deep it goes
- Explains the self-contained container, fixed tensor layouts, K-quant superblocks, and importance-calibrated i-quant codebooks.
- Examples
- Uses llama-2-7b Q4_K_M and Q8_0 files, reporting sizes of 3.80 GB and 6.67 GB.
- What you can do after
- Build CPU-ready GGUF artifacts by converting Hugging Face checkpoints and quantizing them with llama.cpp tools.
Domain-Specific Small Language Models
A model packaged in the GGUF format for local inference with llama.cpp-style runtimes.
- How it is used
- Packages quantized models for distribution and use in llama.cpp-style local inference workflows.
- How deep it goes
- Compares GGUF with GGML through packaging and metadata, without detailing the format specification.
- Examples
- Uses Ollama and LM Studio for local GGUF inference and names llama.cpp’s conversion script for custom models.
- What you can do after
- Operate GGUF-quantized models locally through Ollama or LM Studio, including offline inference.
Read how Domain-Specific Small Language Models explains GGUF →
Also covered in
What to know first
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- C++C++ is a statically typed, general-purpose programming language that supports procedural, object-oriented, generic, and low-level systems programming.
- ONNXONNX (Open Neural Network Exchange) is an open standard and format for representing and exchanging machine-learning models across frameworks, tools, and runtimes.
- serverA server is a computer system or software process that provides data, resources, or services to other systems, typically by responding to requests.
Related concepts
- metadata sectionhas part · A metadata section is a distinct portion of a data representation that stores descriptive information associated with the represented content.
- llama.cppproduced by · llama.cpp is an open-source C/C++ software project for running large language models efficiently across a range of hardware.
- Ollamaused by · Ollama is software for downloading, managing, running, and serving language models locally.
Questions about GGUF
What is GGUF?
GGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.
Which Manning books explain GGUF?
LLMs in Production, Build Applications with Local AI Models on a Mac, Quantization and Fast Inference and Domain-Specific Small Language Models.
What should I know before learning GGUF?
Manning books that teach GGUF usually introduce parameter, C++, ONNX and server first.
What are the parts of GGUF?
GGUF includes gguf header, metadata section, packed tensor data and tensor descriptor table.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



