Rate limiting restricts how much work a client can start over a period or at one time. It protects APIs from overload caused by legitimate spikes, accidental request floods, and malicious traffic. The goal is not to accept every request at any cost; it is to preserve service capacity and acceptable latency for work that can still be completed. section 4.5.4 section 5.3.1
A request-rate limit and a concurrency limit address different failure modes. A per-user token bucket gives each client tokens that refill steadily; each request consumes a token, and requests are rejected when the bucket is empty. A concurrent-request limit instead bounds work already in progress, preventing retrying or slow clients from accumulating expensive active requests. section 4.5.4 section 5.3.1
Use rate limits as part of an overload-control design rather than as an isolated endpoint rule:
- Identify expensive endpoints and the clients or identities whose work should be bounded. section 4.5.6
- Choose a rate bound for request floods and a concurrency bound for clients that hold many slow or retrying requests open. section 4.5.4
- Keep shared, persistent client state across service instances. Redis can increment a per-client counter for a time window and expire it automatically, at the cost of a lookup on every request. section 4.5.4
- Measure queue depth and concurrent requests, then use admission control to reject or delay work when thresholds indicate the service cannot meet acceptable latency. section 5.3.1
- Test the same limits used in production so developers encounter limit behavior before launch. section 4.5.4
Load shedding complements rate limiting during a system-wide incident. Rate limits protect the service from particular clients; load shedders protect critical operations when aggregate demand exceeds capacity. Classify traffic by priority, then drop lower-priority operations first so essential work can continue. This turns overload into an explicit policy decision instead of letting queues grow until requests time out. section 4.5.4 section 5.3.1
Layered API protection
A payment API can first limit requests per user with a token bucket, then cap each user's concurrent requests. If an incident still puts the system under stress, it can shed lower-priority reads of historical data before rejecting critical charge-creation requests. The layers distinguish accidental client flooding, retry cascades, and system-wide capacity loss. section 4.5.4 section 5.3.1
The limiter must not become a new single point of failure. A fail-open design lets requests pass when limiter code fails or Redis is unavailable, avoiding an outage caused by the protection mechanism itself. That choice trades some temporary protection for continued availability and should be considered alongside the risk of unbounded load. section 4.5.4
Operationally, rate limiting belongs in scale planning. Review whether expensive endpoints have limits, decide which operations can be dropped during spikes, categorize background jobs by urgency, and load-test beyond normal capacity. Runbooks should include load-shedding procedures, while service calls need timeouts so failures surface quickly instead of hanging. section 4.5.6 section 5.3.1


















