liveBook · Manning

Batching

As explained in Rearchitecting LLMs

Request batching is the processing of multiple inference requests together as a single batch.

Grouping inference requests by task lets one active adapter process a batch before switching. This avoids per-request adapter changes that can negate performance benefits. section 7.3.6

Figure 7.7 Adapter routing pattern for multiple specializations over a single base model. A task router classifies incoming requests and buffers them into separate queues. A switch selects which adapter is active: the model processes one full batch, swaps the adapter, and processes the next. The base model stays in memory throughout — only the lightweight adapter changes between batches. The dashed outline indicates the inactive adapter waiting for its turn.

A task router queues requests by type; the model processes each queue as a batch with its corresponding adapter, then switches. section 7.3.6

How Rearchitecting LLMs covers it

How deep it goes
Routes requests into task queues, runs each batch with one active adapter, then switches; it does not quantify batching gains.
Examples
Uses task-type queues to process one adapter-specific batch before switching to the next.
What you can do after
Operate production inference by grouping requests by task and scheduling each batch under its corresponding active adapter.
How it connects
Coordinates adapter routing with LoRA, DoRA, and a shared base model.

Also explained in

Compare how 23 books explain batching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.