Batching accumulates multiple items and processes them in one operation. It improves throughput and reduces network overhead, but requires explicit limits for batch size, buffering, flushing, and failure recovery. section 5.3.6 section 7.5.3
For embedding ingestion, group chunk texts and send them together rather than making one API call per chunk. A batch size of 100 is a reasonable default, while provider rate limits and average chunk length determine tuning. section 5.3.6
A production batching design typically addresses these concerns:
- Accumulate items in memory until the configured batch size is reached. section 7.5.3
- Flush partial batches periodically so low-traffic workloads are not delayed indefinitely. section 7.5.3
- Retry failed batches by returning them to the front of the buffer. section 7.5.3
- Bound memory with a maximum buffer size, dropping the oldest data when necessary. section 7.5.3
- Keep batching outside the request path when publishing telemetry, allowing callers to continue without a network round-trip. section 7.5.3
Embedding ingestion
An EmbeddingGenerator walks through chunks in slices of batch_size, extracts their texts, sends one model-client request per slice, and appends the returned vectors. Query embedding remains a separate single-text operation. section 5.3.6
Listing 7.9 ObservabilityClient batching and buffering
class ObservabilityClient:
def __init__(self, channel,
batch_size: int = 100,
flush_interval_seconds: float = 5.0,
max_buffer_size: int = 10_000):
self._stub = ObservabilityServiceStub(channel)
self._span_buffer: list[Span] = [] #A
self._metric_buffer: list[MetricRecord] = [] #A
self._log_buffer: list[LogEvent] = [] #A
self._batch_size = batch_size
self._max_buffer_size = max_buffer_size
self._flush_interval = flush_interval_seconds
self._flush_thread = threading.Thread( #B
target=self._periodic_flush, daemon=True) #B
self._flush_thread.start()
def end_span(self, span_id: str,
status: StatusCode = StatusCode.OK,
error_message: str = None):
span = self._finalize_span(
span_id, status, error_message)
self._span_buffer.append(span)
if len(self._span_buffer) >= self._batch_size:
self._flush_spans() #C
def _flush_spans(self):
if not self._span_buffer:
return
batch = self._span_buffer[:self._batch_size]
self._span_buffer = (
self._span_buffer[self._batch_size:])
try:
self._stub.RecordSpan( #D
RecordSpanRequest(spans=batch)) #D
except Exception:
if (len(self._span_buffer)
< self._max_buffer_size):
self._span_buffer = ( #E
batch + self._span_buffer) #E
# else: drop oldest data to prevent
# unbounded memory growth
def _periodic_flush(self):
while True:
time.sleep(self._flush_interval)
self._flush_spans() #F
self._flush_metrics() #F
self._flush_logs() #FThe implementation shows separate span, metric, and log buffers; size-triggered and timer-triggered flushing; retry preservation; and bounded memory behavior. section 7.5.3






















