Adaptive request batching

As explained in Designing Deep Learning Systems: A guide for software engineers

Dynamic batching is the runtime aggregation of separately arriving inference requests into batches for joint execution.

Adaptive request batching aggregates separately arriving inference requests at runtime, allowing them to execute jointly. In Triton, each model has a configurable scheduler that can batch requests before passing them to the model backend. section 7.4.3

A production serving path assigns batching responsibilities as follows:

• Accept inference requests through REST or gRPC interfaces. section 7.4.3
• Configure batching independently for each model scheduler. section 7.4.3
• Pass batched inputs to the backend that executes the model and return the outputs. section 7.4.3

Batch-capacity configuration

A Triton model configuration can declare max_batch_size: 8, alongside input and output tensor specifications. The scheduler can use that model-level capacity when forming inference batches. section 7.4.3

Figure 7.10 Triton Inference Server high-level architecture (Source: NVIDIA Developer, https://developer.nvidia.com/nvidia-triton-inference-server)

Use the architecture figure to trace requests from the serving interface through model schedulers and backends to returned outputs. section 7.4.3

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Explains scheduler placement and model-level configuration but stops short of detailing queue timing or batching algorithms.

Examples

Uses Triton’s REST or gRPC request path and a model configuration declaring max_batch_size: 8.

What it assumes

Assumes familiarity with inference requests and model serving, while locating batching within schedulers and backends.

What you can do after

Operate and configure model-level batching in Triton while selecting compatible backends and serving infrastructure.

How it connects

Connects inference serving with model schedulers, model repositories, model backends, and GPU or CPU execution.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning System DesignDynamic batching is an inference-serving technique that combines requests arriving within a configured time window into a batch for execution.

Compare how 2 books explain adaptive request batching →

Related concepts

• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• throughputproduces · The amount of work, data, or output a system processes or completes per unit of time.
• batchingkind of · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
• batch sizeconfigured by · Batch size is the number of items grouped together for processing or computation as a single batch.
• meancontrasts with · A mean is a measure of central tendency, calculated as the arithmetic average of values or, for a random variable, as the probability-weighted average of its possible values.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.