Production is the point where a model becomes useful beyond a laptop: it runs in a live environment, accepts real requests, and performs intended work for end users. For an LLM, that usually means packaging the model as a service or API, then deploying it on suitable infrastructure. section 6.0
A model that passes internal benchmarks can still fail in production. Real workloads expose practical constraints such as excessive size, slow responses, unrepresentative tests, and neglected surrounding systems—including data and deployment infrastructure. Production success therefore depends on engineering the system around the model, not selecting weights alone. section 9.3 section 12.2
A pragmatic production workflow often looks like this:
- Build or select a model, then improve its deployability with quantization or task-specific adaptation such as LoRA and QLoRA. section 9.5
- Package it behind an API and deploy it on infrastructure matched to the required workload, whether a cloud cluster or an edge device. section 6.0
- Test with representative customer interactions, measure the resulting behavior, and iterate before scaling investment in hardware, data, or retrieval systems. section 9.3 section 11.3
A deliberately small proof of concept
An LLM running on a Raspberry Pi can be exposed through a llama.cpp server and accessed through a simple chat window or OpenAI-compatible client. The prototype may run at roughly five tokens per second—or slower—making its limitations visible and supporting a concrete case for more hardware, data, retrieval, or research. section 11.3
$./server -m ./models/llava/llava-v1.6-mistral-7b-q4_k_m.gguf --host $PI_IP_ADDRESS --api-key $API_KEY
Inspect the server command that launches a quantized model with a host address and API key, making the deployment boundary concrete. section 11.3
