High-performance data processing may outgrow a single computer, making clusters and network-aware design necessary. Distributed setups bring challenges such as sharing data across machines, managing remote processes, and dealing with network and filesystem latency. section 1.2.2
Network choices trade performance against reliability and overhead:
- Binary payloads can be more efficient than JSON, and direct TCP sockets can replace HTTP where speed and latency matter. section 1.2.2
- UDP can reduce overhead when the extra reliability of TCP is unnecessary. section 1.2.2
Python supports high-performance data work through its libraries and ecosystem, despite performance costs associated with dynamic typing and garbage collection. A simple way to compare implementations is to time matrix multiplication in CPython, PyPy, and JavaScript. section 1.3
Figure 1.3 API calls via the network stack. Understanding the alternatives available for network communication can dramatically increase the speed of internet-based applications.
Use the network-stack diagram to locate where communication choices can affect application speed. section 1.2.2
