An edge device runs computation near the user or data source rather than depending on a centralized service. For an LLM application, that can make local deployment worth considering when connectivity is poor or an API round trip would hurt the experience. It is a deployment choice, not a guarantee of fast inference. section 1.2.2 section 11.5
A story that travels offline
Imagine a choose-your-own-adventure game that uses a device’s GPS location to shape its plot. Players may travel into mountains or out to sea, where an LLM API may be unreachable. Running the model on their device avoids making the experience depend on that connection, though deploying LLMs this way remains exploratory. section 1.2.2
On a small device such as a Raspberry Pi, the practical work is fitting and running the model:
- Reduce memory use where possible, including with a Lite operating system. In a memory-limited environment, quantization is required for inference. section 11.5
- Without a hardware accelerator, compile the model for the device’s architecture with a tool such as llama.cpp. section 11.5
- Test the result, not just whether it starts: inference may still be too slow, and lower quantization raises perplexity. Remote setup also has mundane hurdles, such as finding the Pi’s IP address. section 11.5
Choose local deployment for a concrete need, such as unreliable connectivity or latency-sensitive interaction. Otherwise, an API can simplify integration while introducing network latency and service-reliability risks. Either choice should serve the application’s user experience; getting a model deployed is often more work than preparing it. section 1.2.2 section 8.0 section 11.5
