A queue stores incoming work items until a worker processes them, usually in arrival order. In model serving, requests enter an inference queue while a background worker collects and processes them. section 17.1.4
Batching starts when the queue is full or when its oldest request reaches a maximum waiting time. A thread-safe queue lets the API continue accepting requests while inference runs. section 17.1.4
Restaurant orders
Treat requests as kitchen tickets: collect enough tickets, or wait until one has waited too long, then process them together while new orders continue arriving. section 17.1.4
