Network bandwidth is the maximum rate at which data can move through a storage or communication channel. In data analysis, it measures how much data enters or leaves a system per unit of time. When transfers dominate runtime, improving the transfer path can matter more than adding processor capacity. section 7.1 section 7.1.3
When faster disks beat faster processors
A cloud migration caused processing times to increase dramatically as data volumes grew. If parallel tasks must all read from one file, CPU upgrades may produce little improvement; replacing the hard drive with a faster one may deliver a larger gain. section 7.1 section 7.1.3
Use bandwidth as part of a broader performance investigation:
- Measure where the workload spends time and energy before optimizing calculations. section 7.1.3
- Compare bandwidth with latency, throughput, parallelism, response time, availability, scalability, and reliability. section 7.1
- Check whether disk operations, rather than CPU capacity, limit the workload. section 7.1.3
def monitor_cpu_io_memory(duration, interval):
cpu_percentages = []
io_counters = []
memory_usages = []
end_time = time.time() + duration
while time.time() < end_time:
cpu_percentages.append(psutil.cpu_percent(interval=interval))
io_counters.append(psutil.disk_io_counters())
memory_usages.append(psutil.virtual_memory().used)
time.sleep(interval)
return cpu_percentages, io_counters, memory_usages
# Monitor CPU, I/O activity, and memory usage for 10 seconds with
1-second intervals
cpu_percentages, io_counters, memory_usages =
monitor_cpu_io_memory(duration=10, interval=1)
# Capture memory usage before running your_function()
memory_before = psutil.virtual_memory().used
def your_function():
# Your code here
pass
# Run your function
your_function()
# Capture memory usage after running your_function()
memory_after = psutil.virtual_memory().used
# Calculate function's memory usage
function_memory_usage = memory_after - memory_before
# Analyze CPU usage
avg_cpu_usage = sum(cpu_percentages) / len(cpu_percentages)
print(f"Average CPU usage: {avg_cpu_usage}%")
# Analyze I/O activity
read_bytes = [c.read_bytes for c in io_counters]
write_bytes = [c.write_bytes for c in io_counters]
print(f"Total read bytes: {sum(read_bytes)}")
print(f"Total write bytes: {sum(write_bytes)}")
# Analyze memory usage
avg_memory_usage = sum(memory_usages) / len(memory_usages) - memory_before
print(f"Average memory usage during function execution:
{avg_memory_usage / (1024 * 1024):.2f} MB")
print(f"Function memory usage:
{function_memory_usage / (1024 * 1024):.2f} MB")
A Python monitoring workflow samples CPU, disk I/O, and memory while a function runs, helping identify whether transfer activity is constraining performance. section 7.1.3
Average CPU usage: 1.54% Total read bytes: 2028772421120 Total write bytes: 2388311421440 Average memory usage during function execution: 2.59 MB Function memory usage: 1.32 MB
Sample monitoring output reports CPU usage, total bytes read and written, and memory consumption for comparison during diagnosis. section 7.1.3
