liveBook · Manning

GGUF

GGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.

How 4 Manning books explain it

LLMs in Production

GGUF is a model-file format designed to store quantized language models for efficient local loading and inference.

How it is used
Enables local, CPU-oriented model loading and inference on edge devices, including Raspberry Pi systems and phones.
How deep it goes
Explains conversion, tokenizer packaging, file management, and q4_K_M quantization, but defers detailed quantized-format coverage.
Examples
Uses a converted LLaVA model, q4_K_M quantization, and a Raspberry Pi network service as practical examples.
What you can do after
Build it from scratch by converting and quantizing a model for local service through llama.cpp.

Read how LLMs in Production explains GGUF →

Build Applications with Local AI Models on a Mac

A file format for storing machine-learning model tensors and metadata for inference, including quantized models.

How it is used
Packages fine-tuned models for local inference in Ollama.
How deep it goes
Explains quantization tradeoffs and outlines export and deployment without detailing GGUF file internals.
Examples
Compares quantization levels and illustrates a 7B model’s approximate memory requirements at different precisions.
What you can do after
Apply it via Unsloth and Ollama to export, create, and run a custom model locally.

Read how Build Applications with Local AI Models on a Mac explains GGUF →

Quantization and Fast Inference

GGUF is a file format for packaging quantized machine-learning models and their associated metadata for inference runtimes.

How it is used
Packages quantized models for CPU-first inference with llama.cpp, including deployments where GPU hardware is unavailable.
How deep it goes
Explains the self-contained container, fixed tensor layouts, K-quant superblocks, and importance-calibrated i-quant codebooks.
Examples
Uses llama-2-7b Q4_K_M and Q8_0 files, reporting sizes of 3.80 GB and 6.67 GB.
What you can do after
Build CPU-ready GGUF artifacts by converting Hugging Face checkpoints and quantizing them with llama.cpp tools.

Read how Quantization and Fast Inference explains GGUF →

Domain-Specific Small Language Models

A model packaged in the GGUF format for local inference with llama.cpp-style runtimes.

How it is used
Packages quantized models for distribution and use in llama.cpp-style local inference workflows.
How deep it goes
Compares GGUF with GGML through packaging and metadata, without detailing the format specification.
Examples
Uses Ollama and LM Studio for local GGUF inference and names llama.cpp’s conversion script for custom models.
What you can do after
Operate GGUF-quantized models locally through Ollama or LM Studio, including offline inference.

Read how Domain-Specific Small Language Models explains GGUF →

Also covered in

What to know first

Related concepts

Questions about GGUF

What is GGUF?

GGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.

Which Manning books explain GGUF?

LLMs in Production, Build Applications with Local AI Models on a Mac, Quantization and Fast Inference and Domain-Specific Small Language Models.

What should I know before learning GGUF?

Manning books that teach GGUF usually introduce parameter, C++, ONNX and server first.

What are the parts of GGUF?

GGUF includes gguf header, metadata section, packed tensor data and tensor descriptor table.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.