liveBook · Manning

Batching

As explained in Designing AI Systems

Batching is the accumulation of multiple items so they can be processed or transmitted together.

Batching accumulates multiple items and processes them in one operation. It improves throughput and reduces network overhead, but requires explicit limits for batch size, buffering, flushing, and failure recovery. section 5.3.6 section 7.5.3

For embedding ingestion, group chunk texts and send them together rather than making one API call per chunk. A batch size of 100 is a reasonable default, while provider rate limits and average chunk length determine tuning. section 5.3.6

A production batching design typically addresses these concerns:

  • Accumulate items in memory until the configured batch size is reached. section 7.5.3
  • Flush partial batches periodically so low-traffic workloads are not delayed indefinitely. section 7.5.3
  • Retry failed batches by returning them to the front of the buffer. section 7.5.3
  • Bound memory with a maximum buffer size, dropping the oldest data when necessary. section 7.5.3
  • Keep batching outside the request path when publishing telemetry, allowing callers to continue without a network round-trip. section 7.5.3

Embedding ingestion

An EmbeddingGenerator walks through chunks in slices of batch_size, extracts their texts, sends one model-client request per slice, and appends the returned vectors. Query embedding remains a separate single-text operation. section 5.3.6

Listing 7.9 ObservabilityClient batching and buffering
class ObservabilityClient:
    def __init__(self, channel,
                 batch_size: int = 100,
                 flush_interval_seconds: float = 5.0,
                 max_buffer_size: int = 10_000):
        self._stub = ObservabilityServiceStub(channel)
        self._span_buffer: list[Span] = []    #A
        self._metric_buffer: list[MetricRecord] = []    #A
        self._log_buffer: list[LogEvent] = []    #A
        self._batch_size = batch_size
        self._max_buffer_size = max_buffer_size
        self._flush_interval = flush_interval_seconds
        self._flush_thread = threading.Thread(    #B
            target=self._periodic_flush, daemon=True)    #B
        self._flush_thread.start()
 
    def end_span(self, span_id: str,
                 status: StatusCode = StatusCode.OK,
                 error_message: str = None):
        span = self._finalize_span(
            span_id, status, error_message)
        self._span_buffer.append(span)
        if len(self._span_buffer) >= self._batch_size:
            self._flush_spans()    #C
 
    def _flush_spans(self):
        if not self._span_buffer:
            return
        batch = self._span_buffer[:self._batch_size]
        self._span_buffer = (
            self._span_buffer[self._batch_size:])
        try:
            self._stub.RecordSpan(    #D
                RecordSpanRequest(spans=batch))    #D
        except Exception:
            if (len(self._span_buffer)
                    < self._max_buffer_size):
                self._span_buffer = (    #E
                    batch + self._span_buffer)    #E
            # else: drop oldest data to prevent
            # unbounded memory growth
 
    def _periodic_flush(self):
        while True:
            time.sleep(self._flush_interval)
            self._flush_spans()    #F
            self._flush_metrics()    #F
            self._flush_logs()    #F

The implementation shows separate span, metric, and log buffers; size-triggered and timer-triggered flushing; retry preservation; and bounded memory behavior. section 7.5.3

How Designing AI Systems covers it

How deep it goes
Specifies batch sizing, periodic flushing, retries, and memory bounds, but does not develop provider-specific optimization formulas.
Examples
Uses chunk embedding batches and buffered spans, metrics, and logs as its concrete implementation examples.
What it assumes
Assumes familiarity with chunks, model clients, telemetry records, provider limits, and temporary service unavailability.
What you can do after
Build it from scratch using chunk slices, bounded buffers, threshold-triggered flushes, periodic delivery, and retry preservation.
How it connects
Connects embedding generation with rate limits and telemetry delivery with buffering, retries, asynchronous flushing, and observability.

Also explained in

Compare how 23 books explain batching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.