Batch dispatch groups concurrent audio streams so a speech-recognition model can process requests together rather than separately. Deep Speech 2 (DS2) used it to make GPU-based cloud transcription more practical when the model was too large for on-device use. section 4.4.3
Even batches of two to four streams could reuse loaded model weights, reducing memory-bandwidth overhead. DS2 often achieved 50–70 millisecond latency under moderate load; later accounts describe median latency below 70 milliseconds. These figures show the deployment goal, not a guarantee for every workload. section 4.4.3 section 4.4.4
