A custom metric exposes application-defined measurements that default service metrics cannot capture. Use one to monitor behavior important to the business, support alerting, or report changes in model outcomes. section 11.1.1
Build a custom metric into a deployed inference service by:
- Installing prometheus-client and defining a Prometheus metric with a name, documentation, and suitable value buckets or counters. section 11.1.2
- Updating the service logic to record each relevant observation, such as calling observe with an inference confidence score. section 11.1.2
- Serving the application, checking the /metrics endpoint, and plotting the exposed values in Grafana through Prometheus. section 11.1.2
Object-detection confidence
An object-detection service records confidence scores in decile buckets from 0 to 1. A histogram reveals score distributions and can expose periods of poor image quality or degrading model performance. Grafana can visualize the data and calculate a 90th percentile with histogram_quantile. section 11.1.2
Business behavior counters
A recommendation service can count requests where ranked movies are present or absent, then display those counters as a Grafana line chart. section 11.1.2
confidence_histogram = Histogram(
name="confidence_score",
documentation="The confidence score of the prediction",
buckets=(
0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1
),
)
Inspect the Histogram definition that names the confidence metric, documents its purpose, and assigns decile buckets. section 11.1.2
