Rate limiting protects model serving systems from request surges by placing requests in a queue and restricting processing frequency or concurrency. section 4.4.2
A practical rate-limiting design:
- Check incoming requests against the configured limit. section 4.4.2
- Process only the permitted requests concurrently and hold the remainder in the queue. section 4.4.2
- Apply stricter limits to anonymous users and higher limits after login to support monitoring and auditing. section 4.4.2
Two-request concurrency limit
Four model serving requests arrive. Two execute immediately under the limit; the other two wait until those requests finish. section 4.4.2
Figure 4.17 A flowchart of four model serving requests sent to the model serving system. However, only two are under the current rate limit, which allows a maximum of two concurrent model serving requests. Once the system has finished processing those two requests, it will proceed to the remaining two requests in the queue.
Visualizes queue admission, concurrent processing, and deferred requests. section 4.4.2
Figure 4.18 A comparison of behaviors from different rate limits applied to authenticated and unauthenticated users
Compares rate-limit behavior for authenticated and unauthenticated users. section 4.4.2
