liveBook · Manning

Cold-start behavior

As explained in AI Model Evaluation

The behavior of a recommendation system when users or items have little or no interaction history.

Cold-start behavior is the unstable early performance of a model before it has accumulated feedback or its serving environment has settled. Adaptive recommenders may need learning interactions, while other systems may wait for indexes, embeddings, caches, or precomputed features to populate. Early engagement can therefore reflect stabilization rather than durable model quality. section 6.5.3

Handle cold-start behavior as an experiment-design and diagnostic problem:

  • Define a stabilization period using elapsed time, interaction volume, or both; exclude that period from the primary analysis or report it separately. section 6.5.3
  • Use longer warm-up requirements for adaptive or high-variance systems, and verify that the remaining sample can still detect the expected effect. section 6.5.3
  • Diagnose recommendation cold starts with slices containing sparse-history users or new items, while checking that each slice has enough data for a responsible conclusion. section 3.6.2
  • Continue monitoring after warm-up because feature drift, novelty effects, latency, and infrastructure reliability can change online results. section 6.8

A recommendation experiment

A team might treat the first three days or 100,000 treatment impressions, whichever comes later, as warm-up. It then evaluates the recommendation model on later traffic and separately inspects sparse-history users and new items rather than treating early aggregate engagement as final evidence. section 6.5.3 section 3.6.2

How AI Model Evaluation covers it

How deep it goes
Explains causes, warm-up criteria, planning ranges, and diagnostic slices, but treats timing ranges as sanity checks rather than rules.
Examples
Uses warm-up days or impression thresholds and sparse-history-user or new-item slices for recommendation evaluation.
What it assumes
Assumes experiment metrics and recommendation data containing interaction history, item metadata, outputs, and relevance information.
What you can do after
Evaluate recommendation experiments by separating stabilization effects from post-warm-up performance and checking representative cold-start slices.
How it connects
Connects cold-start behavior with A/B testing, online learning, caching, retrieval, embeddings, feature drift, and offline evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.