Deployment makes a model available for inference in a target environment. For language models, the choice depends on memory needs, hardware, and where the model will run. section 4.2 section 11.0
Deployment decisions include:
- Estimate inference memory from the model’s parameter count and chosen precision; the estimate includes an average 20% overhead. section 4.2
- Choose an environment and serving approach: options include vLLM, FastAPI, and on-device deployment with MLC LLM. section 11.0
- For private, offline Android deployment, PyTorch alternatives include MLC LLM; the workflow involves model conversion and compilation. section 11.4 section 11.4.1
Memory estimates
A 2.7-billion-parameter GPT-Neo model is estimated at 12.96 GB in FP32 and 5.4 GB in FP16. A Mistral 7B model is estimated at about 33.6 GB in FP32 and 16.2 GB in FP16. section 4.2
An MLC LLM Android deployment proceeds from local build to installation:
- Install prerequisites, then build the runtime and selected model libraries. section 11.4.1
- Customize and build the template Android app, bundle it with model weights, and install the APK and weights on the target phone. section 11.4.1
- Check target hardware, benchmark performance, and optimize the model as needed. section 11.4.1
