Chapter 11: Deploying an LLM on a Raspberry Pi: How low can you go?

Introduction

The chapter introduces a hands-on project to run a locally accessible LLM service on a Raspberry Pi, using its limitations to teach practical deployment lessons.

• Raspberry Pi deployment roadmap: Describe the planned project: set up a Raspberry Pi on a local network, convert and quantize a model to GGUF, and serve it as an OpenAI GPT drop-in replacement that other devices on the home network can query.
  Why it matters: This roadmap connects the device setup, model preparation, and serving steps into one deployment project.
• Edge deployment learning tradeoffs: Explain why this is a production-only learning project rather than a recommended enterprise deployment: the Pi-hosted LLM is expected to be slower, less efficient, and less accurate, but exposes the difference between what is possible and what is useful and teaches principles that can later scale up.
  Why it matters: Realistic expectations make the project's shortcomings useful lessons rather than surprises.

Key terms: Raspberry Pi, LLM Service, GGUF, quantizing, drop-in replacement, OpenAI GPT model, production-only project, possible and useful, limitations, scale up

11.1 Setting up your Raspberry Pi

Introduces the Raspberry Pi LLM server project, its hardware requirements, practical caveat, and setup-to-serving path.

• Raspberry Pi project requirements: State the hardware needed to follow the project: a single-board computer with at least 8 GB of RAM, a MicroSD card of at least 32 GB, and a power supply. Describe when a network cable or MicroSD adapter is also needed.
  Why it matters: These requirements determine whether you can follow the setup and run the chosen model.
• Pi server setup path: Trace the project from installing an operating system on the Pi, through installing packages and libraries and preparing the LLM, to serving generated text to computers on the home network.
  Why it matters: This sequence connects initial device setup to the working LLM service.
• Deployment purpose and limits: Explain that serving an LLM on a Raspberry Pi is feasible but is presented primarily as a demonstration rather than a generally recommended deployment choice. Note that other boards or smaller Pis may work, though the instructions use a Raspberry Pi 5 with 8 GB of RAM.
  Why it matters: It sets realistic expectations about following the project and choosing different hardware.

Key terms: Raspberry Pi 5 8 GB, MicroSD, power supply, ethernet cable, operating system (OS), packages and libraries, LLM server, home network, single-board computer

11.1.1 Pi Imager

The section walks through using Pi Imager to put a lightweight OS on a microSD card and configure the Pi for network and SSH access before its first boot.

• Lightweight operating system choice: Explain why the project uses Raspberry Pi OS Lite 64-bit and how to find it in Pi Imager after selecting the Pi device.
  Why it matters: A lightweight OS leaves more of the Pi’s limited RAM available for running the model.
• MicroSD imaging and boot: Describe how to select the Pi device, OS, and microSD storage target in Pi Imager, then write the OS and boot the Pi from the card. Emphasize the warning against selecting the computer’s main drive.
  Why it matters: Selecting the right storage target both prepares the Pi to boot and avoids writing over the wrong drive.
• First boot access settings: Describe the OS customizations needed before imaging: set a hostname and non-default username and password, arrange Wi-Fi or ethernet access, and enable SSH with password authentication in the Services tab.
  Why it matters: Network access and SSH are necessary for the subsequent steps to reach and configure the Pi.

Key terms: Pi Imager, Raspberry Pi OS Lite 64-bit, Raspberry Pi OS (Other), microSD, storage device, image, OS customizations, hostname, Wi-Fi, ethernet

11.1.2 Connecting to Pi

Shows how to find a Raspberry Pi’s network address and connect to its terminal from a laptop using SSH, without keeping a monitor and keyboard attached.

• Finding the Pi address: Describe how to find the Pi’s IP address through the router’s list of connected devices or, with a monitor and keyboard attached, by running ifconfig or ip a and looking for inet.
  Why it matters: You need the Pi’s address before your laptop can connect to it.
• Identifying a headless Pi: Explain how to identify the Pi when neither router access nor an extra monitor is available: inspect network information from the laptop, turn the Pi off, and compare the results to see which address disappears; check again after turning it on because its address may change.
  Why it matters: This provides a fallback when the easier ways of identifying the Pi are unavailable.
• Remote terminal connection: Describe using ssh with the Pi’s username and IP address, accepting the first-connection host fingerprint prompt, and entering the password set in Pi Imager to reach the Pi’s terminal from the laptop.
  Why it matters: SSH lets the Pi act as a small server without a dedicated monitor or keyboard.

Key terms: IP address, router, ifconfig, ip a, inet, ipconfig, arp -a, nslookup, SSH, username

11.1.3 Software installations and updates

This section prepares a connected Raspberry Pi to run an LLM by installing dependencies, building llama.cpp, and downloading the model files.

• System and repository setup: Describe updating the Pi, installing git and pip, and cloning llama.cpp. Explain why checking out the tested commit can prevent problems caused by changes to the repository.
  Why it matters: These steps establish a repeatable starting point for the rest of the deployment.
• Isolated environment and native build: Explain that the section creates and activates a Python virtual environment, installs requirements, and builds llama.cpp because its C++ components must be compiled for the hardware. Distinguish the simple Make build from using CMake when an environment needs more specific build settings.
  Why it matters: The software must be installed and compiled for the machine that will run it.
• LLaVA model file download: State that the section uses huggingface-cli to download Llava-v1.6-Mistral-7B along with its tokenizer and configuration files before moving on to make the model usable on a small device.
  Why it matters: A built runtime still needs the model and its accompanying files.

Key terms: git, pip, llama.cpp, git commit, virtual environment, requirements, C++, Make, CMake, Llava-v1.6-Mistral-7B

11.2 Preparing the model

This section prepares a downloaded model for Raspberry Pi serving by converting it to GGUF, quantizing it to fit the device, and offering ways past common setup problems.

• Single-file GGUF conversion: Explain how the downloaded safetensor model and tokenizer information are converted with convert.py into a single GGUF checkpoint that the repository’s C++ code can use. Describe why the original files may need to be removed or moved off the Pi afterward.
  Why it matters: Conversion makes the model usable by the serving code, but keeping both copies can strain microSD storage.
• Quantization for Pi constraints: Describe quantizing the converted GGUF model to q4_K_M to reduce its size, and relate the resulting little-over-4-GB 7B model to the 8-GB Raspberry Pi’s memory constraints.
  Why it matters: The smaller quantized model is what makes serving this 7B model practical on the Pi.
• Model preparation recovery options: Describe the section’s troubleshooting options: redownload a possibly corrupted model, rebuild dependencies or reboot, try the steps on a better-equipped computer, or use an already quantized model to keep moving.
  Why it matters: These options help distinguish a bad download or build from the Pi’s hardware limitations and provide a route past a stalled preparation.

Key terms: safetensor, tokenizer, GGUF, convert.py, .gguf checkpoint, quantization, q4_K_M, 7B parameter model, llama.cpp, make clean

11.3 Serving the model

This section shows how to access a llama.cpp model on a Raspberry Pi and uses the slow but quick-to-build deployment to illustrate the value of a rapid proof of concept.

• Local model service access: Describe how to reach the llama.cpp service from another computer on the local network using the Pi’s IP address and the server’s API key, and explain why startup may take time.
  Why it matters: These details let you tell a slow model load from a service that is ready to answer requests.
• OpenAI client redirection: Explain how an application using OpenAI’s Python package can query the Pi service by changing its base_url to the Pi’s address and supplying the server’s API key. Distinguish the model name used in the example from the model actually running.
  Why it matters: An existing client can talk to the local service without switching Python packages, but its familiar model name can be misleading.
• Browser chat and edge speed: Describe how to use the server’s built-in minimal GUI in a browser and what response speed the section says to expect on the Raspberry Pi.
  Why it matters: Trying the browser interface makes the hardware’s performance limits immediately visible.
• Rapid prototype production workflow: Justify building a quick proof of concept on constrained hardware before negotiating what a production project needs. Explain how a working but limited demo provides evidence for further investment and builds stakeholder trust.
  Why it matters: A fast demonstration can make the case for the hardware, data, or other work needed to build something customers will keep using.

Key terms: llama.cpp, Pi’s IP address, API key, local network, OpenAI’s Python bindings, base_url, gpt-3.5-turbo, built-in minimal GUI, port 8080, five tokens per second

11.4 Improvements

This section shifts from a guided Raspberry Pi walkthrough to ways readers can adapt the project for themselves.

• Beyond the guided walkthrough: Explain why the project does not end with following the prescribed commands: adapting it, such as by choosing a different model or tooling, is a further step toward learning and production use.
  Why it matters: The walkthrough gets the project running; modifications help you make it your own.

Key terms: guided assistance, different model, different tooling

11.4.1 Using a better interface

The section suggests improving the Raspberry Pi LLM project’s user experience by trying interfaces and servers beyond llama.cpp.

• llama.cpp and interface alternatives: Distinguish llama.cpp’s strengths in compiling and quantizing from its more convenience-oriented serving and frontend features, and describe how other tools can extend the project’s interface.
  Why it matters: Choosing a tool for the interface need not mean replacing the tool that prepared the model.
• User experience through frontends: Explain why adding a frontend can make the project more engaging, while recognizing that the named tools are options to try rather than required choices.
  Why it matters: An interface changes how users experience the model, not just how developers query it.

Key terms: llama.cpp, compiling, quantizing, serving, frontend, KoboldCpp, SillyTavern, customization, frontend and server

11.4.2 Changing quantization

This section explains how to choose a llama.cpp quantization format when a Raspberry Pi or other hardware cannot comfortably run the model used earlier in the project.

• Quantization size and quality tradeoff: Explain how llama.cpp quantization formats differ in bit level, resulting model size, and RAM requirements, and why more aggressive quantization generally reduces quality and raises perplexity.
  Why it matters: The smallest model file is not automatically the best choice if its output quality suffers.
• Small Pi model choices: Describe the options for running an LLM on a 4 or 6 GB Pi: choose a smaller-parameter model, quantize more aggressively, and expect that it may run more slowly.
  Why it matters: Limited memory calls for changing the model or its representation rather than abandoning the deployment outright.
• Hardware memory limits: Explain that larger hardware still imposes model-size limits, including VRAM as well as RAM when using GPU frameworks, and that reducing other memory use can leave room for a larger model.
  Why it matters: The same memory-budget reasoning applies beyond Raspberry Pi deployments.

Key terms: llama.cpp, quantization formats, q4_K_M, perplexity, 4 or 6 GB Pi, 7B, 1B, 3B, Q2_k, Q3_K_S

11.4.3 Adding multimodality

The section adds image input to a LLaVA model served on a Raspberry Pi, shows how to query it, and explains why the result is impractically slow.

• LLaVA image projection requirement: Explain why the existing LLaVA server cannot accept images until a multimodal projection file is added: the converted model includes the language portion, but images must be encoded for the language model to use them.
  Why it matters: The projection file is what makes image input usable by the already-trained LLaVA model.
• Multimodal serving and requests: Describe how to serve the prepared model with the projection file and send an image through a multimodal chat request using the OpenAI-compatible API.
  Why it matters: Both server-side image handling and an image-bearing request are needed to test the model.
• Raspberry Pi multimodal latency: Explain why successfully serving LLaVA on a Raspberry Pi does not make it practical to use: image encoding alone can take dozens of minutes before text generation begins.
  Why it matters: A working deployment can still be too slow to provide a useful experience.
• Unstable multimodal server support: Describe how llama.cpp’s multimodal server support was removed after problems with its initial implementation, and why that change makes reproducing or running the example risky.
  Why it matters: A deployment can break when an immature dependency changes unexpectedly.

Key terms: LLaVA, multimodal projection file, vision portion, .gguf, --MMPROJ, mmproj.gguf, OpenAI API, encode_image, messages, Raspberry Pi

11.4.4 Serving the model on Google Colab

The section shows how to move the Raspberry Pi prototype to Google Colab for testing and combine llama.cpp Python bindings, an OpenAI-compatible API, and LangChain for an extensible RAG deployment.

• Colab testing tradeoffs: Explain that most model-serving steps carry over from the Raspberry Pi to Google Colab’s free tier, apart from Pi setup and small changes for a Jupyter environment. Describe why Colab is useful for testing but not a source of unlimited production GPU capacity.
  Why it matters: This helps you choose a low-cost place to debug without mistaking free GPU access for production infrastructure.
• Python bindings and API integration: Describe how llama.cpp Python bindings can replace repeatedly downloading the repository and running Make, with installation options for hardware acceleration. Explain how the example connects a server for the reader’s own model to LangChain’s ChatOpenAI through an OpenAI-compatible API, presenting it as GPT-3.5-turbo.
  Why it matters: This connection makes the local model easier to use through familiar Python and API tooling.
• RAG inference additions: Trace the optional RAG additions in the example: use a sentence transformer to embed a dataset, create and load a FAISS index for inference, prepare a RAG prompt, and use tiktoken to avoid exceeding the context length.
  Why it matters: These additions show how the served model can use retrieved data while respecting its input limit.
• Small hardware deployment scope: Explain the section’s claim that inference and RAG can run on a Raspberry Pi for a prototype, while more compute helps and single-board RAM compute is not a practical answer to serving hundreds of customers.
  Why it matters: Separating prototype feasibility from customer-scale capacity keeps the deployment claim realistic.

Key terms: Google Colab, Jupyter, free tier, GPU, llama.cpp Python bindings, OpenAI API, LangChain, ChatOpenAI, GPT-3.5-turbo, sentence transformer

11.5 Summary

The summary recaps the setup, memory, performance, and serving tradeoffs involved in deploying an LLM on a Raspberry Pi.