Concurrency measures how well a storage layer serves multiple compute engines, users, and pipelines at the same time. In an Iceberg lakehouse, request spikes, read contention, and throughput throttling can turn storage into a bottleneck even when engines use caching and in-memory execution. section 5.1.1
Evaluate concurrency against the workload’s access pattern and performance priority:
- Interactive dashboards need low-latency, consistent performance across many small concurrent reads. section 5.5.1
- Batch ETL can tolerate latency spikes when sustained throughput keeps multi-hour processing within its operating window. section 5.5.1
- Frequent appends and compactions may stress systems optimized for sequential reads rather than concurrent random writes. section 5.1.1
Dashboard traffic
Dozens of users query Iceberg tables during business hours. Elastic scalability or deterministic low-latency retrieval can absorb demand, while a flexible system may require tuning to maintain interactive performance. section 5.5.1
Figure 5.1 Factors to consider in understanding a storage layer’s performance
Use the performance-factor figure to relate concurrency to latency, throughput, consistency, access patterns, file distribution, and compression. section 5.1.1
