A latency degradation experiment measures how slower response times affect users and product outcomes. In this controlled A/B test, a control group receives no added delay while test groups experience different delays. The results help teams make evidence-based decisions about slower models, features, and performance improvements. section 4.4 section 4.4.1
Design the experiment around the user experience, not latency alone:
- Establish a baseline for end-to-end response time and break it down by stages such as feature retrieval, inference, network transfer, and rendering. section 4.4.2
- Add controlled delays to test variants and retain a no-delay control. Keep infrastructure and resources close to production conditions so resource constraints are not mistaken for the effects of the injected delay. section 4.4.1
- Measure system performance alongside user and product metrics. Page-load percentiles show how the experience changed; clicks, abandonment, and downstream consumption show how users responded. section 4.4.2 section 4.4.3
- Limit exposure and duration, define guardrail metrics and stop conditions in advance, and obtain stakeholder approval. The test intentionally worsens the experience for some users. section 4.4
- Use the results to decide whether to optimize a slower feature, accept the trade-off against business goals, or prioritize performance improvements. section 4.4.1
Movie recommendations
A recommendations page has a current p99 of 900 ms. An experiment injects delay in the personalization backend closest to the client, with a control and five delayed variants. It tests whether slower loading reduces clicks, scrolling, and movie consumption. Track p50, p95, and p99 load times, click-through rate, bounce rate, abandonment, and downstream consumption; replicate the test by device type because latency effects may differ. section 4.4.3
Table 4.4 Latency Degradation Test Results on Mobile
|
Variant Cell
|
Added Latency
|
Avg Page Load Time
|
CTR
|
Bounce Rate
|
|
Control
|
0ms
|
1050ms
|
23.5%
|
27.8%
|
|
Test Variant 1
|
100ms
|
1155ms
|
22.7%
|
29.1%
|
|
Test Variant 2
|
200ms
|
1250ms
|
20.2%
|
32.7%
|
|
Test Variant 3
|
300ms
|
1360ms
|
16.4%
|
39.5%
|
|
Test Variant 4
|
400ms
|
1455ms
|
12.2%
|
48.3%
|
|
Test Variant 5
|
500ms
|
1552ms
|
8.7%
|
56.9%
|
Compare the mobile variants: as added delay rises from 0 to 500 ms, average page load time and bounce rate increase while click-through rate falls. section 4.4.3
The measured response helps locate latency levels that may threaten product goals, but the decision still requires judgment: technical speed alone does not establish whether a feature delivers enough value to justify a slower experience. section 4.4 section 4.4.1 section 4.4.3
