liveBook · Manning

Caching

As explained in AI Model Evaluation

The storage of previously computed or retrieved data so subsequent requests can be served more quickly.

Caching stores frequently requested or previously computed data so later requests can respond faster. In AI products, it supports fast, reliable experiences when latency affects search, recommendations, feeds, chat, or checkout. section 4.1 section 4.1.1

Use caching as part of a broader performance strategy:

  • Reduce time to first byte by serving reusable responses or retrieved features without repeating expensive work. section 4.2.1
  • Hide backend delay when results load asynchronously or progressively, while measuring both real and perceived latency. section 4.1.1
  • Treat cached responses as an architectural optimization rather than a substitute for reliable model serving. section 4.1

Recommendation loading

A recommendation row may appear quickly from cached results while newer personalization work continues in the background. Evaluate whether the experience improves visible latency, not merely whether backend inference becomes faster. section 4.1 section 4.1.1

Table 4.1 Model-Serving Lifecycle Metrics
Metric
What It Measures
Why It Matters
How To Improve
Fully Loaded Time
Total time for page & all AI content to load.
Reflects full user-perceived loading experience.
Lazy loading, optimize asset delivery, pre-compute predictions
Model Load Time
Time taken to load the AI model into memory or environment.
Important for startup performance and responsiveness.
Model size reduction, efficient loading methods
Initialization Time
Time for model or system to initialize before inference.
Affects responsiveness after deployment or scale-up.
Warm starts, caching, optimized initialization
Warm-up Time
Time taken for the model to reach optimal performance after idle. This only applies for models and systems that have an initial setup or “cold start” cost before they can deliver fast inference consistently.
Can cause slower responses initially after periods of inactivity
Keep model “warm” with background requests
Cold Start Time
Time taken for the model to start from scratch.
Critical in serverless or autoscaling environments
Minimize dependencies, faster bootstrapping

Use the model-serving lifecycle metrics to place caching alongside warm starts, initialization, and fully loaded time when diagnosing performance. section 4.2.1

How AI Model Evaluation covers it

How deep it goes
Connects caching to real and perceived latency but stops short of explaining cache invalidation or storage internals.
Examples
Uses cached recommendation rows, hidden backend work, TTFB optimization, and cached asynchronous AI content as examples.
What it assumes
Assumes basic familiarity with AI features, model inference, latency, and user-facing product performance.
What you can do after
Apply caching to reduce visible response time and diagnose whether latency originates in inference or the surrounding system.
How it connects
Ties caching to latency, asynchronous loading, progressive rendering, pre-computation, and model-serving performance.

Also explained in

Compare how 49 books explain caching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.