Profiling answers a practical question: where does a data analytics system spend time or resources, and did an optimization actually improve it? It collects detailed execution information, such as time spent in each function or memory allocated, to reveal resource-intensive hot spots. section 7.1
Start by understanding system requirements and choosing measurements that match them. Execution time is a useful starting point, but throughput and resource utilization can also describe system efficiency. section 7.1
Use profiling as a repeatable comparison rather than a one-off stopwatch reading:
- Profile under expected loads instead of relying on an isolated test server. section 7.1.1
- Run the process many times and calculate at least minimum, maximum, and average execution times. section 7.1.1
- Compare the revised system with the previous version to check whether changes produced the desired impact. section 7.1
Profiling bootstrap confidence intervals
A bootstrap confidence-interval calculation repeatedly samples data and recalculates statistics, making it a useful workload to measure. Test several DataFrame sizes, time the grouped calculation, and plot execution time against input size. The resulting pattern helps identify how processing cost changes as the dataset grows. section 7.1 section 7.1.1
import numpy as np
import pandas as pd
import timeit
import matplotlib.pyplot as plt
# Your bootstrap_CI function here
# Your calculate_ci function here
# Use your actual filtered_df DataFrame
def measure_performance(data_sizes):
"""
Measure the performance of the calculate_ci function for different
input data sizes.
Parameters:
data_sizes (list): A list of data sizes to test.
Returns:
list: A list of execution times for each data size.
"""
times = []
for data_size in data_sizes:
subset_df = filtered_df.sample(n=data_size, replace=False,
random_state=42)
grouped_data = subset_df.groupby('seller_state')
start_time = timeit.default_timer()
ci_df = grouped_data.apply(calculate_ci)
end_time = timeit.default_timer()
times.append(end_time - start_time)
return times
data_sizes = [100, 500, 1000, 5000, 9944]
times = measure_performance(data_sizes)
plt.plot(data_sizes, times, marker='o')
plt.xlabel('Size of filtered_df')
plt.ylabel('Execution Time (s)')
plt.title('Performance of calculate_ci Function')
plt.grid(True)
plt.show()
Inspect a Python workflow that samples different input sizes, times a grouped confidence-interval calculation with timeit, and returns execution times for comparison. section 7.1.1
Treat a single execution-time result cautiously: it applies only to its input, system load, and configuration. Concurrent activity can distort measurements, so repeated tests under realistic conditions provide a more reliable performance picture. section 7.1.1
