Cache miss ratio measures the proportion of model lookup requests whose requested model is absent from the model cache. In a multitenant prediction service, it indicates how often model swapping may be required before serving a request. section 6.3.2
Use the metric when managing constrained model-serving resources:
- Apply least recently used eviction to remove models and make room for requested models. section 6.3.2
- Partition models across service instances to reduce misses and make model swapping less disruptive. section 6.3.2
Use the prediction-service architecture to locate the model cache within a multitenant serving workflow. section 6.3.2
