12 Delivering Proven Results with a Four-Bit LLM Server and an On-Device Vision Pipeline
This chapter covers
- Serving a four-bit LLM on a single GPU through vLLM
- Gating a deployment with offline tests before it ships
- Standing up serving observability with Prometheus and DCGM
- Verifying quantized output against a full-precision reference
- Sizing an on-device vision pipeline to a sub-second budget
- Comparing tokens per second, memory, and quality across settings
Shipping a quantized model is not the same as proving it fast. A benchmark can show the forward pass runs in a millisecond. It cannot show that the artifact loaded on the right hardware, ran through the low-precision kernel it was built for, held the latency a user actually feels, and kept answering close to its full-precision reference once real traffic arrived. A speed number measured in isolation proves none of that.
Two habits close that gap. You prove the model before it ships, and you watch it after it goes live. The offline gate fails the build on errors a benchmark never checks, like a wrong metric name, an unsafe telemetry port, an unpinned image, or a figure drawn from data that was never measured. Live observability then catches what tests cannot predict, like latency drift, queue buildup, thermal throttling, and quality that slips after deployment.