liveBook · Manning

Resource utilization

As explained in AI Model Evaluation

Hardware utilization is the proportion of available processor capacity consumed by a running system.

Resource utilization describes how much available hardware capacity a running system uses. CPU and GPU usage, memory footprint, and network activity help assess the infrastructure needed to deliver model predictions. These measures do not directly assess model quality, but they matter for cost, responsiveness, and deployability. section 2.1.1 section 4.4.2

Evaluate utilization alongside end-to-end response time, model inference time, application load time, and throughput. Measure a latency baseline and break it down by request stage before choosing optimizations. Looking across frontend, backend, model, and infrastructure metrics helps identify where changes will most improve the user experience. section 4.4.2 section 4.2.4

Accuracy is not enough

A model that returns results in 1,500 ms is not viable for a product with a 400 ms latency SLA, however accurate it is. Pair quality evaluation with engineering constraints before deciding whether to deploy it. section 2.1.1

In an A/B test, track CPU and GPU utilization as infrastructure guardrails alongside user-facing success metrics. Also watch inference latency, throughput, cost per request, and errors. A model that raises click-through rate by 2% but doubles GPU cost should not be treated as an unquestioned win. Work with infrastructure and platform teams when heavier models or request-path changes may affect system health. section 6.7.1

Optimization involves trade-offs. Reducing model complexity may lower inference latency but also reduce prediction quality; increasing batch size may improve throughput while worsening latency for individual users. Judge changes against product quality and system requirements together. section 4.2.4

How AI Model Evaluation covers it

How deep it goes
Identifies utilization metrics and evaluation guardrails, but does not specify a measurement implementation.
Examples
Illustrates deployment constraints with a 1,500 ms model against a 400 ms SLA and an A/B test with doubled GPU cost.
What you can do after
Evaluate and understand utilization alongside quality, latency, throughput, and cost when deciding whether a model is deployable.
How it connects
Connects utilization with inference latency, throughput, model quality, cost per request, and A/B testing.

Also explained in

Compare how 13 books explain resource utilization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.