Queuing latency is the time a request spends waiting in a queue before processing resumes. It differs from processing latency, when work is actively done. Both can contribute to the request’s total response time, so faster processing alone may not remove a delay caused by waiting. section 2.4 section 1.1
Consider an HTTP handler waiting for a database result. Rather than burn CPU in a busy-poll loop, the application can sleep and yield the CPU, improving throughput and energy efficiency. When the result arrives, another process may still be running. The application then waits for the OS scheduler to run it again, adding queuing latency even though the database has responded. section 2.4
Queues can appear inside other components too. A network interface card places arriving packets in receive queues before the operating system network stack and application handle them. The full delay from packet arrival to its availability in a userspace thread is packet processing latency; queue wait is only one possible part of that path. section 1.1 section 2.4
To investigate a slow request:
- Separate time spent waiting in a queue from time spent actively processing. Where queue entry and processing start can be observed, their difference gives the queue wait. section 2.4
- Check scheduling delays and queues in the components the request uses before assuming its active work is the bottleneck. section 2.4
- Account for sequence: delays in successive stages compound, so removing one queue wait need not eliminate the rest of the response time. section 2.4
