liveBook · Manning

Edge computing

As explained in LLMs in Production

A computing architecture that processes data near its source or point of use to reduce dependence on remote services.

An immersive LLM experience cannot depend on a remote API when users have poor service or no internet access. Edge computing moves inference onto a device near the user. It can also avoid the network round trip and third-party availability problems that matter in latency-sensitive applications. Running an LLM on an edge device is possible, but remains exploratory. section 1.2.2

The immediate constraints are memory and speed. An edge device with 8 GB of RAM and no GPU may struggle to deliver even one token per second; there is no uniformly good fix. Edge devices can support ML development and inference, but training an LLM on one is not a practical target here. section 6.4

Choose hardware and model format together:

  • A Raspberry Pi can gain inference capacity from a Coral USB TPU, though USB 3.0 bandwidth limits it. TPU use requires converting the model and inference code to TensorFlow; Optimum can convert models to TensorFlow Lite format. section 6.4
  • A Jetson-style board includes a GPU and CUDA, avoiding the separate accelerator purchase and the TensorFlow requirement. ExecuTorch is another offering for edge inference. section 6.4

Use edge deployment when connectivity or response time makes an API unsuitable, but budget for the device’s memory, compute, and conversion requirements. Quantization and other parameter-efficient methods are part of the surrounding effort to make local LLM use feasible; they do not erase the hardware trade-offs. section 1.2.2 section 6.4

How LLMs in Production covers it

How deep it goes
Compares device constraints, accelerators, and model conversion paths, but stops short of a worked edge deployment.
Examples
Uses an offline immersive experience and an 8 GB Raspberry Pi with a Coral USB TPU to expose practical constraints.
What it assumes
Assumes basic familiarity with LLM inference and APIs while spelling out device memory, speed, and accelerator constraints.
What you can do after
Evaluate/understand when edge LLM inference merits exploration and which hardware and conversion constraints shape the choice.
How it connects
Links edge inference to API serving, latency, quantization, TensorFlow Lite, TPUs, GPUs, and CUDA.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

Compare how 5 books explain edge computing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.