GGUF
GGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.
LLMs in Production
GGUF is a model-file format designed to store quantized language models for efficient local loading and inference.
- How it is used
- Enables local, CPU-oriented model loading and inference on edge devices, including Raspberry Pi systems and phones.
- How deep it goes
- Explains conversion, tokenizer packaging, file management, and q4_K_M quantization, but defers detailed quantized-format coverage.
- Examples
- Uses a converted LLaVA model, q4_K_M quantization, and a Raspberry Pi network service as practical examples.
- What you can do after
- Build it from scratch by converting and quantizing a model for local service through llama.cpp.
Build Applications with Local AI Models on a Mac
A file format for storing machine-learning model tensors and metadata for inference, including quantized models.
- How it is used
- Packages a fine-tuned, quantized model for local inference with Ollama.
- How deep it goes
- Explains the memory–quality tradeoff, compares quantization tags, and walks through export and deployment without detailing file internals.
- Examples
- Shows quantization tags, an Unsloth GGUF export call, and a Modelfile pointing to a local GGUF.
- What you can do after
- Apply it via Unsloth and Ollama to export a fine-tuned model as GGUF and run it locally.
Read how Build Applications with Local AI Models on a Mac explains GGUF →
Domain-Specific Small Language Models
A model packaged in the GGUF format for local inference with llama.cpp-style runtimes.
- How it is used
- Packages quantized models for distribution and offline inference through llama.cpp-style local runtimes.
- How deep it goes
- Defines single-file packaging and metadata improvements, but defers detailed conversion and quantization procedures.
- Examples
- Uses Ollama conversion and configuration, plus LM Studio and Jan desktop model management.
- What you can do after
- Operate/configure quantized models for offline inference using local runtimes and model-management tools.
Read how Domain-Specific Small Language Models explains GGUF →
Related concepts
- Ollamaused by · Ollama is software for downloading, running, and serving language models locally.
- llama.cppused by · llama.cpp is an open-source C/C++ software project and runtime for running large language models efficiently on local hardware.
- Modelfileconfigured by · A Modelfile is Ollama’s declarative configuration file for specifying a model’s base, prompts, parameters, and other settings.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.