liveBook · Manning

Rate limit

As explained in Designing AI Systems

Rate limiting is the controlled restriction of how many requests or tokens a system accepts from a client, workflow, or organization over a period of time.

Rate limiting is a platform-side guardrail that restricts request or token consumption before traffic reaches model providers. It protects budgets and shared capacity from runaway jobs, bot attacks, load tests, and fallback chains that would otherwise shift spending between providers. section 3.6 section 1.1.1

Choose limits according to the resource and failure mode being controlled:

  • Per-user limits prevent one user from monopolizing capacity. section 3.6
  • Per-workflow limits prevent one application from starving other workflows of shared provider access. section 3.6
  • Global limits impose an organization-wide ceiling across users and workflows. section 3.6
  • Request limits control call volume; token limits account for expensive, large-context requests. section 3.6
  • Concurrency limits restrict how many requests can run simultaneously, complementing rate limits during traffic spikes. section 3.6 section 1.1.1
Listing 3.14 RateLimitConfig dataclass for organizational protection
@dataclass
class RateLimitConfig:
    requests_per_minute: int = None        #A
    tokens_per_minute: int = None          #B
    requests_per_day: int = None           #C
    max_concurrent_requests: int = None    #D
    scope: RateLimitScope = RateLimitScope.WORKFLOW  #E

Use the configuration model to express request, token, daily, concurrency, and scope constraints in one platform boundary. section 3.6

How Designing AI Systems covers it

How deep it goes
Defines request, token, daily, concurrency, and scope dimensions but does not specify enforcement algorithms or storage.
Examples
Uses per-user, per-workflow, and global scopes alongside request-per-minute, token-per-minute, daily, and concurrency settings.
What it assumes
Assumes a platform model service that mediates provider access and distinguishes application-side from platform-side responsibilities.
What you can do after
Operate/configure organizational consumption guardrails through a platform-level model service.
How it connects
Connects rate limiting to provider rate limits, fallback chains, concurrency, cost control, and platform responsibilities.

Also explained in

Compare how 19 books explain rate limit →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.