liveBook · Manning

GGUF

As explained in Build Applications with Local AI Models on a Mac

A file format for storing machine-learning model tensors and metadata for inference, including quantized models.

GGUF is a file format used to run quantized models on consumer hardware. In a local fine-tuning workflow, it packages the trained weights and tokenizer so Ollama can load the result. GGUF is not the training tool: a tool such as Unsloth trains and exports the model; Ollama serves it locally. section 16.6.2 section 16.6.4 section 17.7.1

Quantization compresses a model so it needs less memory. For scale, a 7B model’s weights may take about 28 GB at 32-bit precision, versus roughly 4–5 GB in a 4-bit quantized file. Running it also needs memory for context and runtime overhead. More compression can reduce answer quality, particularly on subtle reasoning or specialized tasks. section 16.6.2

A practical route from your examples to a local model is:

  • Prepare input/output examples, choose a base model, and fine-tune it with a framework such as Unsloth. section 16.6.3
  • Export the trained result as a quantized GGUF file. Check that your model architecture, training method, and GGUF export are supported. section 16.6.4
  • Point a Modelfile at the exported file, create the model with `ollama create`, then run and test it with Ollama. section 16.6.3 section 17.7.1

Choosing a quantization level

Tags such as `q4_K_M` and `q5_0` identify quantization levels. A lower number generally means a smaller memory requirement but lower quality. For quick tests, `q2_K` is very small; `q4_0` balances size and quality; `q5_0` needs more RAM for higher quality. Use `ollama show gemma3:4b` to inspect a model’s quantization. section 16.6.2

model.save_pretrained_gguf("my_model", tokenizer, quantization_method="q4_k_m")

This Unsloth export call saves a GGUF model using the `q4_k_m` quantization setting. section 16.6.4

cat << 'EOF' > Modelfile                                  
FROM ./my_finetuned_model.gguf

SYSTEM "You are a specialized assistant trained on my domain data."

PARAMETER temperature 0.7
PARAMETER num_ctx 4096
EOF

This Modelfile points `FROM` at a local GGUF file and sets the system prompt and parameters Ollama will use. section 17.7.1

How Build Applications with Local AI Models on a Mac covers it

How deep it goes
Explains the memory–quality tradeoff, compares quantization tags, and walks through export and deployment without detailing file internals.
Examples
Shows quantization tags, an Unsloth GGUF export call, and a Modelfile pointing to a local GGUF.
What it assumes
Assumes training examples and a selected base model while explaining quantization through memory use and quality tradeoffs.
What you can do after
Apply it via Unsloth and Ollama to export a fine-tuned model as GGUF and run it locally.
How it connects
Connects GGUF to quantization, fine-tuning, Unsloth, Ollama, and Modelfiles.
Cover of Build Applications with Local AI Models on a Mac
liveBookmanning.com
Read Build Applications with Local AI Models on a Mac on liveBook Get the book at manning.com

Also explained in

Compare how 3 books explain GGUF →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build Applications with Local AI Models on a Mac. The section links point to the parts of the book it draws on.