Cold-start behavior is the unstable early performance of a model before it has accumulated feedback or its serving environment has settled. Adaptive recommenders may need learning interactions, while other systems may wait for indexes, embeddings, caches, or precomputed features to populate. Early engagement can therefore reflect stabilization rather than durable model quality. section 6.5.3
Handle cold-start behavior as an experiment-design and diagnostic problem:
- Define a stabilization period using elapsed time, interaction volume, or both; exclude that period from the primary analysis or report it separately. section 6.5.3
- Use longer warm-up requirements for adaptive or high-variance systems, and verify that the remaining sample can still detect the expected effect. section 6.5.3
- Diagnose recommendation cold starts with slices containing sparse-history users or new items, while checking that each slice has enough data for a responsible conclusion. section 3.6.2
- Continue monitoring after warm-up because feature drift, novelty effects, latency, and infrastructure reliability can change online results. section 6.8
A recommendation experiment
A team might treat the first three days or 100,000 treatment impressions, whichever comes later, as warm-up. It then evaluates the recommendation model on later traffic and separately inspects sparse-history users and new items rather than treating early aggregate engagement as final evidence. section 6.5.3 section 3.6.2
