Reliability is the ability of an inference system to keep performing its required functions during unexpected load spikes, model bugs, or hardware failures. Choosing the cheapest platform can create greater business risk when an outage arrives at the wrong time. section 15.1
Assess reliability as part of production inference design:
- Compare platform cost with failure consequences rather than selecting hardware by price alone. section 15.1
- Ask how the system will respond to unexpected demand, model defects, and hardware failure. section 15.1
- Balance reliability against latency, throughput, scalability, cost, flexibility, and security instead of optimizing one factor in isolation. section 15.6
The hot-season outage
A low-cost inference platform may look efficient until a sudden failure occurs during a high-demand season. Paying more for a time-tested solution can be justified when the operational and financial consequences of failure outweigh the initial savings. section 15.1
