Compute cost is the resource expenditure required to run a model or system operation. Inference cost deserves attention before implementation: infrastructure can scale with demand until it exceeds the revenue generated by the system. Device-side inference can reduce that growth, although model delivery and related services still add expenses. section 15.1
Evaluate compute cost alongside the constraints that shape a serving system:
- Estimate how infrastructure cost changes as request load increases, rather than judging hardware only by its purchase price. section 15.1
- Compare model accuracy with latency, throughput, and cost; larger models may improve accuracy, but the benefit eventually saturates. section 15.2.1
- Inspect architecture and platform together: models with the same parameter count can have different parallel-computation demands. section 15.2.1
- Treat reliability as part of the cost decision, because cheaper hardware can create greater operational losses during failures or demand spikes. section 15.1
- Prefer a practical balance over either indiscriminate model expansion or choosing the simplest model solely to minimize cost. section 15.2.1
Balancing overloaded inference devices
A system running on low-end GPUs used a small model for concurrent requests, but the GPUs failed under load. Routing later requests to CPUs through a mixed-device session pool balanced utilization while meeting latency requirements. section 15.1
Figure 15.1 An example of wide versus deep models. While they have the same number of parameters, the deep one is more limited in terms of parallel computation
Compare wide and deep models with equal parameter counts to see why architecture can change parallel-computation cost. section 15.2.1
