Llama.cpp
llama.cpp is an open-source C/C++ software project and runtime for running large language models efficiently on local hardware.
LLMs in Production
llama.cpp is an open-source software project for running and deploying large language models locally, especially on commodity hardware.
- How it is used
- Deploys and serves LLMs locally on edge devices with constrained resources.
- How deep it goes
- Covers installation, reproducible checkout, environment preparation, compilation, quantization, serving, and frontend integration.
- Examples
- Uses Raspberry Pi deployment, fixed Git commits, CMake or Make builds, local model downloads, and frontend tools such as Ollama and KoboldCpp.
- What you can do after
- Operate/configure a locally compiled, quantized LLM server on constrained hardware.
Build a Large Language Model (From Scratch)
A C/C++ library for efficiently running large language models, used here as the engine behind Ollama.
- How it is used
- Supports local LLM inference for automatically evaluating instruction-fine-tuned model responses.
- How deep it goes
- Defines its implementation language and deployment role, but stops short of explaining internal algorithms or architecture.
- Examples
- Uses `ollama run llama3`, a local chat API, and Python scoring code as concrete integration examples.
- What you can do after
- Operate Ollama to run a local Llama 3 model and generate evaluation responses.
Read how Build a Large Language Model (From Scratch) explains llama.cpp →
Related concepts
- Ollamaimplemented by · Ollama is software for downloading, running, and serving language models locally.
- GGUFuses · GGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.