Rate limiting controls how many requests a client can make within a specified interval. It is especially important in front of an expensive LLM, where uncontrolled traffic can exhaust resources, disrupt service, or deny other users fair access. section 6.1.4
Choose an algorithm according to the traffic pattern and fairness requirements:
- Fixed window permits a fixed number of requests per interval, making it simple to configure but vulnerable to bursts at window boundaries. section 6.1.4
- Sliding-window logs count requests within a moving lookback period, reducing boundary effects and distributing traffic more evenly. section 6.1.4
- Token buckets spend tokens per request and block requests when the bucket is empty, supporting controlled bursts. section 6.1.4
Protecting an LLM API
A FastAPI service combines access-key validation with SlowAPI rate limiting. The limiter keys requests by remote address, while an exception handler returns a rate-limit response when a client exceeds its allowance.
Listing 6.2 Example API with access keys and rate limiter
from fastapi import FastAPI, Depends, HTTPException, status, Request
from fastapi.security import OAuth2PasswordBearer
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address
from slowapi.errors import RateLimitExceeded
import uvicorn
api_keys = ["1234567abcdefg"] #1
API_KEY_NAME = "access_token"
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="token")
limiter = Limiter(key_func=get_remote_address)
app = FastAPI()
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
async def get_api_key(api_key: str = Depends(oauth2_scheme)):
if api_key not in api_keys:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid API Key",
)
@app.get("/hello", dependencies=[Depends(get_api_key)])
@limiter.limit("5/minute")
async def hello(request: Request):
return {"message": "Hello World"}
Inspect the complete API listing for a concrete access-control and rate-limiting configuration. section 6.1.4
