liveBook · Manning

Token limit

As explained in AI Governance

A token cap is a configured maximum on the number of language-model tokens that may be processed or generated for an interaction or session.

A token limit caps token usage for a request or session. It can be enforced before a model call in middleware or applied as a per-request or per-session quota. section 2.4.1 section 4.3.4

Use token limits as one layer of resource control:

  • Pair quotas with per-user or API-key rate limits, timeouts, and hard provider spend limits; budget alerts alone do not stop resource use. section 4.3.4
  • Monitor token consumption and cost, and consider circuit breakers that pause execution before costs escalate. section 4.3.4

Runaway requests

A QA script sent thousands of requests in a loop and incurred several hundred dollars in charges. Per-session token caps were added alongside rate limits and hard spend limits. section 4.9

How AI Governance covers it

How deep it goes
Specifies where limits apply and related safeguards, but does not describe token-counting mechanics.
Examples
Illustrates accidental request loops causing high charges and attacks that trigger excessive output or extended reasoning.
What you can do after
Operate token quotas alongside rate limits, timeouts, and spend controls to constrain resource consumption.
How it connects
Pairs token limits with rate limits, timeouts, input validation, hard spend limits, and circuit breakers.

Also explained in

Compare how 12 books explain token limit →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.