Batching groups incoming LLM requests briefly before dispatching them. When traffic surges, an application can use a queue to control concurrency and smooth demand instead of starting every request immediately. The trade-off is a short wait while requests collect; under heavy load, throughput and overall response times can improve. section 9.2.2
Choose the batching mechanism to match the workload:
- For interactive traffic, queue requests in the application. Drain up to a target batch size on a repeating interval, then dispatch their model calls concurrently. Each query remains a separate Chat Completion call; this does not bundle multiple prompts into one endpoint request. section 9.2.2
- For large jobs that can wait, use the OpenAI Batch API instead. It processes offline-style API work at lower cost but with longer turnaround; the illustrated submission specifies a 24-hour completion window. The two approaches can coexist. section 9.2.2
Holiday-sale customer support
An online store’s support chatbot receives bursts of product, refund, and policy questions during a sale. An asyncio.Queue holds each query with its callback and streaming preference. A background processor takes up to two requests per cycle, creates a task for each, awaits them together, and then sleeps for one second. The request handler separately chooses a model and either streams a reply or retrieves a cached answer. Batching manages the burst; semantic caching and model fallback address repeated questions and expensive calls. section 9.2.5 section 9.2.5
async def batch_processor(batch_size: int = 2, interval: float = 1.0):
"""
Periodically collects up to batch_size requests from request_queue,
processes them in one go, then sleeps 'interval' seconds before repeating.
This reduces overhead spikes when many users query at once.
"""
while True:
batch = []
try:
while len(batch) < batch_size:
item = request_queue.get_nowait()
batch.append(item)
except asyncio.QueueEmpty:
pass
if batch:
tasks = []
for r in batch:
tasks.append(asyncio.create_task(handle_request(r)))
await asyncio.gather(*tasks)
await asyncio.sleep(interval)
The processor shows the operational boundary: drain a limited number of queued items, launch one task per item, await the group, and pause before checking again. section 9.2.5
