Llama.cpp
llama.cpp is an open-source C/C++ software project for running large language models efficiently across a range of hardware.
LLMs in Production
llama.cpp is an open-source software project for running and deploying large language models locally, especially on commodity hardware.
- How it is used
- Deploys and serves LLMs locally on edge devices with constrained resources.
- How deep it goes
- Covers installation, reproducible checkout, environment preparation, compilation, quantization, serving, and frontend integration.
- Examples
- Uses Raspberry Pi deployment, fixed Git commits, CMake or Make builds, local model downloads, and frontend tools such as Ollama and KoboldCpp.
- What you can do after
- Operate/configure a locally compiled, quantized LLM server on constrained hardware.
Build a Large Language Model (From Scratch)
A C/C++ library for efficiently running large language models, used here as the engine behind Ollama.
- How it is used
- Supports local LLM inference for automatically evaluating instruction-fine-tuned model responses.
- How deep it goes
- Defines its implementation language and deployment role, but stops short of explaining internal algorithms or architecture.
- Examples
- Uses `ollama run llama3`, a local chat API, and Python scoring code as concrete integration examples.
- What you can do after
- Operate Ollama to run a local Llama 3 model and generate evaluation responses.
Read how Build a Large Language Model (From Scratch) explains llama.cpp →
Also covered in
- Quantization and Fast InferenceIn early access (MEAP)
- Domain-Specific Small Language Models
What to know first
- C++C++ is a statically typed, general-purpose programming language that supports procedural, object-oriented, generic, and low-level systems programming.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
Related concepts
- GGUFproduces · GGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.
- Ollamaimplemented by · Ollama is software for downloading, managing, running, and serving language models locally.
- BitNethas part · BitNet is a neural-network architecture family that uses extremely low-bit, often ternary, weight representations to reduce memory and computation costs.
- Cortexused by · A local AI engine for running language models on a user's machine with support for multiple backends and quantized models.
Questions about llama.cpp
What is llama.cpp?
llama.cpp is an open-source C/C++ software project for running large language models efficiently across a range of hardware.
Which Manning books explain llama.cpp?
LLMs in Production and Build a Large Language Model (From Scratch).
What should I know before learning llama.cpp?
Manning books that teach llama.cpp usually introduce C++ and language model first.
What are the parts of llama.cpp?
Llama.cpp includes BitNet.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

