An inference engine loads a machine-learning model and executes it to produce predictions from inputs. Choosing one is a serving decision: the engine, hardware, model shape, latency target, and deployment environment must work together. section 15.3.1
Evaluate an inference engine by checking how it handles the following practical concerns:
- Hardware placement: execution may be split across CPUs, GPUs, and specialized accelerators. CoreML, for example, dynamically divides a graph among the CPU, GPU, and Apple Neural Engine. section 15.3.1
- Inference optimization: ONNX Runtime can remove training-only graph parts and fuse operations, but these optimizers are preparation tools rather than the engine itself. section 15.3.1
- Deployment behavior: serverless inference makes cold-start time, artifact size, model load time, worker limits, and scaling rules part of the engine choice. section 15.3.2
- Model family: LLM inference has distinctive memory and autoregressive constraints, motivating specialized choices such as vLLM, TGI, or GGML. section 15.3.1
A cold-start tradeoff
Suppose a model takes 100 milliseconds when warm but 5,000 milliseconds after an idle period. With processing-time pricing, the cold request costs 50 times more, so a serverless deployment must weigh savings at low loads against latency and initialization overhead. section 15.3.2
Figure 15.3 A wide spectrum of tools with their focus on either research or production serving goals
Use the spectrum to compare tools oriented toward research workflows with those designed for production serving. section 15.3.1
