liveBook · Manning

Infrastructure

As explained in AI Model Evaluation

Infrastructure is the computing, storage, networking, and runtime support required to operate a software system.

Infrastructure supplies the computing, storage, networking, tooling, and runtime support that makes software and model evaluation operational rather than theoretical. In evaluation work, it must support data preparation, logging, training, serving, experimentation, and reliable measurement. section 3.7 section 6.7

Treat infrastructure as part of the evaluation strategy, not as a post-launch concern:

  • Diagnostic offline evaluations require tooling, data infrastructure, and robust logging. section 3.7
  • Online A/B tests require infrastructure to train and serve models, plus an experimentation platform to compare control and test variants. section 6.7
  • Production guardrails track inference latency, throughput, CPU and GPU utilization, cost per request, and downstream error rates. section 6.7.1
  • LLM-as-a-judge pipelines need versioned prompts, token and cost controls, batching, observability, calibration, and reproducible documentation. section 9.10

A model that wins but slows the product

A new model raises click-through rate by 2% but adds half a second to page loading. Users may bounce, engagement may decline, and infrastructure costs may rise. Tracking latency, system capacity, errors, and GPU cost as guardrails can expose the regression before release. section 6.7.1

Reliable infrastructure connects evaluation results to product reality: a model must improve user outcomes without making the system slower, less stable, harder to scale, or too expensive to operate. section 6.7 section 6.7.1

How AI Model Evaluation covers it

How deep it goes
Extends from tooling, data infrastructure, and logging to latency, capacity, costs, versioning, and reproducibility.
Examples
Uses logging, model serving, latency dashboards, GPU-cost guardrails, and versioned prompts as concrete infrastructure artifacts.
What you can do after
Operate/configure evaluation systems by measuring reliability, latency, capacity, cost, observability, and reproducibility alongside user metrics.
How it connects
Connects offline evaluation, A/B testing, experimentation platforms, model serving, observability, and guardrail metrics.

Also explained in

Compare how 11 books explain infrastructure →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.