Adaptive request batching improves online inference by briefly holding arriving API requests, combining them, and executing one batch instead of one prediction at a time. A typical policy waits for a short window—such as 20 milliseconds—or until 32 requests arrive, whichever comes first. This creates a throughput opportunity without turning a responsive service into an offline batch job. section 15.2.2
Design the batching policy around the model and the service promise:
- Tune the accumulation window against client latency: longer waits can form larger batches, while shorter waits keep real-time responses responsive. section 15.2.2
- Group variable-length language-model inputs by size when padding to the longest input would waste computation; accept smaller batches or a longer accumulation window as the tradeoff. section 15.2.2
- Separate real-time and offline batch requirements even when both paths use the same model; configure the real-time path for tighter queue delays. section 15.5.2
Real-time serving with Triton
A serving system can use Nvidia Triton Inference Server behind an application, with HTTP requests accumulated dynamically before inference. For a low-latency real-time path, the dynamic-batching parameter max_queue_delay_microseconds can be kept below 10 milliseconds, while a separate managed job handles larger offline batches. section 15.5.2
