Throughput measures how many predictions or how much data a system processes per unit of time. It matters most when large volumes must be handled quickly, especially in batch workloads. Unlike latency, which emphasizes prompt individual responses, throughput evaluates aggregate processing capacity. section 15.1
Evaluate throughput alongside the requirements that shape an inference system:
- Balance throughput against latency, scalability, cost, reliability, flexibility, and security; these goals can conflict. section 15.1 section 15.6
- Use batching and parallelization when immediate responses are unnecessary and workloads contain large volumes of data. section 15.1 section 15.5.1
- Monitor job duration, rows processed, resource utilization, success rate, and failure rate to identify production bottlenecks. section 15.5.1
Batch demand forecasting
Supermegaretail runs daily, weekly, and monthly forecasts over large datasets. Docker containers orchestrated by AWS Batch process data from S3 on EC2 machines, scale horizontally, and write predictions back to S3. Throughput is prioritized over specialized hardware because the batch workload allows parallel execution. section 15.5.1
