What you will learn in Fast Python: High performance techniques for large datasets
High performance techniques for large datasets
Chapter by chapter
Concepts explained in this book
- parallelismParallelism is the concurrent execution or progress of multiple computational operations, typically by independent portions of work using one or more processing resources.
- CythonCython is a Python-derived programming language and compiler that supports static typing and translates annotated code into C or C++ extension code.
- dataset splitA dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or evaluation.
- performancePerformance is a measure of how effectively a person, process, model, or system achieves its objectives, including qualities such as output quality, speed, and resource efficiency.
- PythonPython is a high-level, general-purpose, dynamically typed programming language with readable syntax and support for multiple programming paradigms.
- DaskDask is an open-source Python library for parallel and distributed computing using task graphs and scalable data structures for large or partitioned workloads.
- Apache ArrowApache Arrow is an open, language-independent specification and software ecosystem for representing and processing columnar data in memory.
- profilingProfiling is the measurement and analysis of a program’s execution behavior and resource use to characterize performance and identify bottlenecks.
- memory usageMemory usage is the amount of computer memory occupied, allocated, or required by a program, system, data structure, or computation during operation.
- data structureA data structure is an organized representation of data and its relationships that supports defined operations for storing, accessing, or modifying it.
- NumPy arrayA NumPy array, or ndarray, is an n-dimensional array object with a defined shape and element data type, supporting efficient numerical computation.
- GPGPUGPGPU is the use of graphics processing units to perform general-purpose computations, often exploiting their parallel processing capabilities.
- computation graphA computation graph is a directed graph whose nodes represent computational operations or values and whose edges represent dependencies between them.
- lazy evaluationLazy evaluation is an evaluation strategy that defers computing an expression until its value is needed, potentially avoiding the computation entirely.
- listA list is an ordered collection of elements arranged in a sequence.
- text chunkA text chunk is a bounded segment of a larger text treated as a unit for processing, storage, or retrieval.
- NumPyNumPy is an open-source Python library for numerical computing, providing multidimensional arrays and operations on them.
- global interpreter lockThe global interpreter lock is a mutual-exclusion mechanism in CPython that permits only one thread at a time to execute Python bytecode within a process.
- data frameA data frame is a two-dimensional tabular data structure organized into labeled rows and columns, often with columns that can contain different types of values.
- viewA view is a representation or access interface that presents selected, transformed, or derived information from underlying data or state.
- NumbaNumba is a Python compiler that uses just-in-time compilation to generate optimized machine code for numerical code, with support for CPU and GPU targets.
- performance optimizationPerformance optimization is the systematic modification of a system’s implementation, configuration, or operation to improve measures such as speed, responsiveness, throughput, or resource efficiency.
- CPythonCPython is the reference implementation of the Python programming language, written primarily in C and providing its interpreter and runtime.
- MapReduceMapReduce is a data-processing model that maps operations over input elements or partitions and combines their intermediate results through reduction or aggregation.
- tensor shapeA tensor shape is an ordered sequence of dimension sizes specifying the extent of a tensor along each axis.
- multithreadingMultithreading is the execution of multiple threads of control within a program, concurrently through interleaving or simultaneously.
- atomic typeAn atomic type is a type whose supported accesses and operations are indivisible with respect to concurrent execution and provide defined synchronization semantics.
- shared memoryShared memory is a memory region that can be accessed by multiple processes, threads, or other concurrent execution units.
- pandaspandas is an open-source Python library providing labeled data structures and tools for manipulating and analyzing tabular and other structured data.
- MapA map is an associative data structure that associates keys with values and supports retrieving a value by its key.
- vectorizationVectorization is representing data or computation in vector form so values can be processed together by numerical operations.
- compressionCompression is the encoding of data or information in a more compact representation to reduce its size while retaining content needed for recovery or use.
- CPU cacheA CPU cache is small, fast memory that stores copies of data or instructions for quicker access by a processor.
- multiprocessingMultiprocessing is a programming technique that executes work in multiple operating-system processes, enabling concurrent or parallel execution across available processor resources.
- process poolA process pool is a managed collection of worker processes that executes submitted tasks concurrently.
- ZarrZarr is a storage format and software ecosystem for chunked, compressed, multidimensional arrays organized in hierarchical groups.
- schedulerA scheduler is a software component that selects, orders, or dispatches tasks for execution and may assign them execution resources according to defined policies or constraints.
- hardwareHardware is the physical equipment and components of a computing system that provide resources for computation and software execution.
- setA set is a collection of distinct elements in which membership, rather than order or duplicate occurrences, defines the collection.
- CuPyCuPy is an open-source Python library providing a NumPy-compatible interface for GPU-accelerated numerical computing.
- GPUA graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and other data-parallel workloads.
- asynchronous programmingAsynchronous programming structures operations so that execution can continue or yield control without waiting synchronously for each operation to complete.
- broadcast operationA broadcast operation is a collective communication operation that distributes data from one participant to all participants in a group.
- missing valueA missing value is an expected data value or observation that is absent, unavailable, unknown, or inapplicable.
- BloscBlosc is a high-performance lossless compression library optimized for binary data and designed to exploit blocking and modern processor capabilities.
- ParquetApache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
- line_profilerline_profiler is a Python profiling tool that measures execution time for individual source-code lines.
- CPUA central processing unit (CPU) is a computer processor that executes program instructions and performs computational operations.
- array programmingArray programming is a programming technique that expresses operations over whole arrays rather than iterating imperatively over individual elements.
- dask.distributeddask.distributed is a Dask package that provides a task scheduler, workers, clients, and distributed execution infrastructure.
- memoryMemory is a storage capability or mechanism for retaining information or computational state for later access or use.
- memoryviewA memoryview is a Python object that exposes a view of a memory buffer containing homogeneous data without copying it.
- strideA stride is the step or spacing between successive positions or elements along a dimension or access pattern.
- np.shares_memorynp.shares_memory is a NumPy function that determines whether two arrays share memory.
- NumExprNumExpr is a Python library that evaluates numerical array expressions using optimized operations and reduced intermediate memory.
- Moore's lawMoore’s law is the empirical observation that the number of transistors on an integrated circuit historically tended to double approximately every two years.
- big dataBig data is data whose volume, velocity, variety, or complexity challenges conventional storage, processing, or analysis methods.
- GPU kernelA GPU kernel is a program or function executed on a graphics processing unit to perform computation, often in parallel across many threads.
- fsspecfsspec is a Python library that provides a uniform interface for accessing filesystems and file-like objects across local and remote storage systems.
- model architectureModel architecture is the structural arrangement of a computational model’s components, connections, and operations that determines how it processes inputs and produces outputs.
- thread blockA thread block is a group of GPU threads scheduled together that can cooperate using shared resources such as shared memory.
- arrayAn array is an ordered collection of elements organized in one or more dimensions and accessible by position or index.
- CUDACUDA is NVIDIA’s parallel computing platform and programming model for performing general-purpose computations on NVIDIA GPUs.
- columnar dataColumnar data is tabular data organized and stored by column rather than by complete row, enabling efficient access to selected fields and data compression.
- data localityData locality describes the relationship between where data is located and where or how it is accessed or processed, including proximity, reuse, and location constraints.
- memory allocationMemory allocation is the process of reserving or assigning computer memory for program data and objects during execution.
- object overheadObject overhead is the memory required to represent an object beyond the memory occupied by its payload or explicitly stored data.
- executorAn executor is a software component or abstraction that accepts tasks and arranges for them to run in an execution environment.
- cuda.jitcuda.jit is a Numba decorator that compiles a Python function for execution as CUDA device code or a GPU kernel.
- concurrencyConcurrency is the structuring or execution of multiple computational activities so their progress overlaps in time, whether or not they run simultaneously.
- Zarr arrayA Zarr array is a persistent multidimensional array whose data and metadata are stored in independently addressable chunks.
- Edholm's lawAn empirical observation that telecommunications data rates tend to double approximately every 18 months.
- exponential growth curveAn exponential growth curve represents a quantity that increases by a constant multiplicative factor over equal increments of its input.
- unstructured dataUnstructured data is information that lacks a fixed, predefined schema or consistently organized fields or records.
- memory boundA workload is memory-bound when its performance is limited primarily by memory access or data-transfer capacity rather than computational throughput.
- basebase is a NumPy array attribute referencing the first array object in the chain from which a view derives.
- in-memory processingIn-memory processing performs computations primarily on data held in main memory rather than repeatedly accessing persistent storage.
- QuadLifeQuadLife is a colored extension of Conway's Game of Life in which live cells can have four color states and additional color-transition rules.
- local networkA local network is a computer network connecting devices within a limited physical or administrative area, such as a home, office, or campus.
- boundary checkA bounds check verifies that an index, access, or operation lies within the permitted limits of a data structure, memory region, or range.
- row iterationRow iteration is the technique of processing a tabular dataset one record at a time through sequential procedural evaluation.
- asynchronous serverAn asynchronous server is a server whose operations can yield while waiting for input or other external events so that other client work can proceed.
- ufuncA ufunc is a NumPy universal function that performs elementwise operations on arrays and supports broadcasting.
- chunk granularityChunk granularity is the amount of data or content assigned to each unit when a larger source is divided into chunks for processing, storage, transfer, or retrieval.
- loopA loop is a programming-language control-flow construct that repeatedly executes a block of instructions according to an iteration rule or continuation condition.
- cloud computingCloud computing is a model for on-demand network access to shared computing resources and services that can be provisioned and scaled as needed.
- transistor densityTransistor density is the number of transistors integrated per unit area of a semiconductor device.
- repartitionRepartitioning is the redistribution of records or items across partitions according to a partitioning rule, potentially changing the number of partitions.
- Plasma serverPlasma is an in-memory object store and server associated with Apache Arrow that enables processes to exchange objects efficiently.
- interprocess communicationInterprocess communication is the exchange of data or coordination signals between independently executing processes through defined mechanisms.
- sequential processingSequential processing is the execution of operations one after another in a defined order, without overlapping their execution.
- persistpersist is a Dask collection operation that starts computation and retains the resulting partitions for reuse in subsequent computations.
- Game of LifeConway's Game of Life is a cellular automaton whose grid cells change state across discrete generations according to local neighborhood rules.
- fancy indexingFancy indexing is selecting multiple array elements or positions with a collection of integer, Boolean, or label-based selectors.
- callbackA callback is a function or other callable supplied or registered with a component for invocation at a specified event, condition, or point in execution.
- picklePickle is Python’s standard-library facility for serializing Python objects to byte streams and reconstructing them from those streams.
- transpositionTransposition is an operation that exchanges a matrix’s rows and columns or, more generally, permutes the axes of a multidimensional array.
- BLASBLAS is a standard interface and collection of low-level routines for efficient vector and matrix operations in numerical computing.
- Intel MKLIntel MKL, now Intel oneMKL, is a library of optimized mathematical and scientific computing routines, including implementations of BLAS and LAPACK.
- OpenBLASOpenBLAS is an open-source optimized implementation of the BLAS and LAPACK numerical linear algebra interfaces.
- execution modelAn execution model is an abstract set of rules describing how a computational system organizes, triggers, orders, and carries out computations.
- sys.getsizeofsys.getsizeof is a Python standard-library function that returns an object's shallow memory size in bytes.
- contiguous memory layoutA contiguous memory layout stores adjacent data elements at adjacent memory addresses without gaps between them.
- cuda.gridcuda.grid is a Numba CUDA intrinsic that computes a thread's global index in a one-, two-, or three-dimensional execution grid.
- cdefcdef is a Cython declaration for defining C-level functions, variables, or types without automatically creating a Python-callable interface.
- larger-than-memory dataLarger-than-memory data is data whose total size exceeds the available main memory for processing it all at once.
- high-performance computingHigh-performance computing is the use of powerful computing systems, often with parallel or specialized processing, to perform computationally intensive workloads.
- UDPThe User Datagram Protocol (UDP) is a connectionless transport-layer protocol that sends datagrams without guaranteeing delivery, ordering, or duplicate suppression.
- horizontal scalingHorizontal scaling is increasing or decreasing system capacity by adding or removing computing nodes or service instances that share the workload.
- row groupA row group is a storage and processing unit containing a subset of rows in a columnar data file.
- generatorA generator is a language construct or computation that produces values incrementally on demand, typically suspending and resuming while preserving its state.
- Dask dataframeA Dask dataframe is a partitioned tabular data structure whose operations are represented as a Dask task graph and whose partitions are typically pandas data frames.
- Python wrapperA Python wrapper is a Python interface that exposes or adapts the operations of an underlying implementation, library, or service.
- bisection algorithmA bisection algorithm repeatedly divides an ordered search range to locate a target or insertion position.
- object caching and reuseObject caching and reuse is a memory-management technique that retains existing objects for repeated use instead of repeatedly creating equivalent objects.
- memory hierarchyA memory hierarchy is an arrangement of storage levels with differing capacities, access speeds, latencies, costs, or persistence.
- concurrent.futuresconcurrent.futures is a Python standard-library module providing high-level interfaces for asynchronously executing and coordinating callable tasks with threads or processes.
- time-space tradeoffA time-space tradeoff is a design choice that exchanges computational time for storage or memory, or storage or memory for computational time.
- wrap-aroundWrap-around is a boundary-indexing technique that connects opposite edges of a finite grid so indexing past one edge continues at the other edge.
- nogilnogil is a Cython function qualifier indicating that a function can execute without holding CPython's global interpreter lock.
- bit arrayA bit array is a compact, indexed data structure whose elements are individually accessible binary values.
- instrumentationInstrumentation is the addition or configuration of mechanisms in software or systems to observe, measure, or record execution and behavior.
- applyIn pandas, apply is a method that applies a callable across Series values or along a DataFrame axis and returns the results.
- iterrowsiterrows is a pandas DataFrame method that yields row index and row data pairs for sequential row traversal.
- itertuplesitertuples is a pandas DataFrame method that yields each row as a tuple, optionally including the row index.
- SeriesA Series is pandas’ one-dimensional labeled data structure that associates values with an index.
- ZstandardZstandard, or ZSTD, is a general-purpose lossless compression algorithm and format with configurable trade-offs between compression ratio and speed.
- benchmarkA benchmark is a defined task, workload, test, dataset, or reference standard used under specified conditions to measure or compare systems or methods.
- cProfilecProfile is a Python standard-library deterministic profiler that records function-call counts and execution-time statistics.
- type inferenceType inference is the automatic deduction of program types from expressions, values, declarations, usage, or contextual constraints without requiring every type to be specified explicitly.
- eager evaluationEager evaluation is an evaluation strategy in which expressions are computed as they are encountered rather than deferred until their values are needed.
- C intC int is an integer type in the C programming language whose size and range depend on the implementation.
- unsigned integerAn unsigned integer is an integer representation whose values are nonnegative, with its representable range determined by its bit width and applicable rules.
- data conversionData conversion is the transformation of a value or data from one type, format, or representation to another, explicitly or implicitly.
- mutationMutation is a change to an existing entity, such as a value, state, data structure, or candidate representation.
- numerical overflowNumerical overflow occurs when an arithmetic result falls outside the range representable by its numeric type or format.
- non-object arrayA non-object array is an array whose elements are stored as raw values of a specific data type rather than as references to general objects.
- data transformationA data transformation is an operation that maps input data or representations to output by changing their values, structure, or form.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
