Big data is data whose volume, velocity, variety, or complexity challenges conventional storage, processing, or analysis methods. Its growth creates a widening gap between the data available for analysis and the computing resources able to process it. section 1.0 section 1.1
Big-data workloads become difficult when they combine:
- Rapidly increasing data volumes that demand more processing resources. section 1.1
- Changing representations, especially unstructured data, which increases computational demands. section 1.1
- Python execution constraints, hardware limits, network behavior, cloud architecture, threading, and CPython’s Global Interpreter Lock. section 1.0
Internet traffic as a data-growth signal
Global internet traffic, measured in petabytes per month, illustrates the expanding amount of data that systems must process. Moore’s law does not guarantee that computing capacity will keep pace with this growth. section 1.1 section 1.1
Figure 1.1 The ratio between Moore’s law and Edholm’s law suggests that hardware will always lag behind the amount of data being generated. Moreover, the gap will increase over time.
Compare Moore’s law with Edholm’s law to see why hardware can lag behind data generation and why system-level optimization matters. section 1.1
