How deeply does Domain-Specific Small Language Models cover GGUF?

Defines single-file packaging and metadata improvements, but defers detailed conversion and quantization procedures.

GGUF

As explained in Domain-Specific Small Language Models

A model packaged in the GGUF format for local inference with llama.cpp-style runtimes.

GGUF is a quantized-model format designed for llama.cpp-style local inference. It replaced GGML as llama.cpp’s supported format and packages a model as one distributable file. section 6.4.2

GGUF’s practical advantages include:

Embedding tokenizer information removes the need for an external JSON file. section 6.4.2
Storing hyperparameters as key–value metadata makes it easier to add fields without breaking existing models. section 6.4.2
Supporting local tools such as Ollama, LM Studio, and Jan enables offline inference on personal devices. section 12.7

Local deployment

Convert a custom model with llama.cpp’s convert_hf_to_gguf.py script, define its templates and parameters in an Ollama Modelfile, then serve it locally through Ollama. Jan instead requires GGUF models and reports hardware compatibility during selection. section 12.7

How Domain-Specific Small Language Models covers it

Cover of Domain-Specific Small Language Models

liveBook manning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

LLMs in ProductionGGUF is a model-file format designed to store quantized language models for efficient local loading and inference.
liveBook manning.com

Build Applications with Local AI Models on a MacA file format for storing machine-learning model tensors and metadata for inference, including quantized models.

Compare how 3 books explain GGUF →

Related concepts

Ollamaused by · Ollama is software for downloading, running, and serving language models locally.
llama.cppused by · llama.cpp is an open-source C/C++ software project and runtime for running large language models efficiently on local hardware.
Modelfileconfigured by · A Modelfile is Ollama’s declarative configuration file for specifying a model’s base, prompts, parameters, and other settings.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.