liveBook · Manning

Cost

As explained in AI Model Evaluation

Cost measures the computational, financial, or resource expenditure required to operate a model or system.

Cost is the financial or resource expenditure required to run a model or system. A model that performs well offline can still be impractical in production if it is too expensive to serve. Track cost alongside latency, reliability, and user outcomes rather than treating model quality as sufficient evidence of readiness. section 4.6 section 9.10

For third-party LLMs, cost per request captures the direct expense of a call. Context size drives both cost and latency, especially as conversations grow. Agent tasks can involve multiple model and tool calls, so track calls per task as well as per-request cost when assessing the system. section 4.2.3

Use cost measurements to guide practical decisions:

  • Set cost per request as an A/B test guardrail, alongside inference latency, throughput, and CPU or GPU utilization. A user-metric improvement should not hide an unacceptable infrastructure regression. section 6.7.1
  • Monitor evaluation pipelines from the start. Log token consumption, average response time, and total evaluation cost using billing APIs or internal counters, and visualize them alongside other operational metrics. section 9.10.4
  • Choose judge models with scale in mind. Token usage and model pricing drive evaluation cost, while latency and instruction-following ability also affect feasibility. section 9.10.5
  • Reserve judge panels for decisions where their added cost is justified, such as launch gates or contested safety judgments. For high-volume, low-stakes checks, a validated single judge may suffice, with panels used for periodic audits. section 10.1.2

A two-tier evaluation

Use a lightweight model for broad comparisons or pairwise ranking, then reserve a higher-end model for final adjudication. Log average tokens per request: even a low-cost call can consume substantial budget when repeated thousands or millions of times. section 9.10.5

How AI Model Evaluation covers it

How deep it goes
Explains cost drivers, tracking practices, guardrails, and model-selection trade-offs without developing a detailed costing formula.
Examples
Illustrates a two-tier judge workflow and an A/B test where a click-through improvement doubles GPU cost.
What you can do after
Operate and configure cost monitoring and guardrails for model serving and evaluation pipelines.
How it connects
Connects cost to latency, context size, A/B testing, and LLM-as-a-judge evaluation.

Also explained in

Compare how 17 books explain cost →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.