liveBookmanning.com

What you will learn in Fast Python: High performance techniques for large datasets

High performance techniques for large datasets

by Tiago Antao

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• parallelismParallelism is the concurrent execution or progress of multiple computational operations or tasks.
• CythonCython is a Python-derived programming language and compiler that supports static typing and translates annotated code into C or C++ extension code.
• dataset splitA dataset split is the division of a dataset into distinct subsets, typically assigned separate roles or distributed for independent storage or processing.
• performancePerformance is the measured effectiveness of a system or model under specified conditions, assessed using relevant qualities such as output quality, speed, responsiveness, throughput, or resource use.
• PythonPython is a high-level, general-purpose, dynamically typed programming language with readable syntax and automatic memory management.
• DaskDask is an open-source Python library for parallel and distributed computing using task graphs and scalable data structures for large or partitioned workloads.
• Apache ArrowApache Arrow is an open, language-independent specification and software ecosystem for representing and processing columnar data in memory.
• profilingProfiling is the measurement and analysis of a program’s runtime behavior and resource use to characterize performance and identify bottlenecks.
• memory usageMemory usage is the amount of computer memory occupied or allocated by a program, process, data, or computation during operation.
• data structureA data structure is an organized representation of data and its relationships that supports defined operations for storing, accessing, or modifying it.
• NumPy arrayA NumPy array, or ndarray, is a multidimensional array object whose elements have a specified data type and that supports efficient numerical computation.
• GPGPUGPGPU is the use of graphics processing units to perform general-purpose computations, often by exploiting their parallel processing capabilities.
• computation graphA computation graph is a directed graph whose nodes represent computational operations or values and whose edges represent data dependencies between them.
• lazy evaluationLazy evaluation is an evaluation strategy that defers computing an expression or value until its result is needed, potentially avoiding the computation entirely.
• listA list is an ordered collection of elements arranged in a sequence.
• text chunkA text chunk is a bounded segment of a larger text or document treated as a unit for processing, storage, or retrieval.
• NumPyNumPy is an open-source Python library for numerical computing that provides multidimensional arrays and operations for working with them.
• global interpreter lockThe global interpreter lock is a mutual-exclusion mechanism in CPython that permits only one thread at a time to execute Python bytecode within a process.
• data frameA data frame is a two-dimensional tabular data structure with labeled rows and columns that can contain values of different types.
• viewA view is an alternate representation or access interface for data derived from or backed by underlying storage without necessarily creating an independent copy.
• NumbaNumba is a Python compiler that uses just-in-time compilation to generate optimized machine code for numerical code, with support for CPU and GPU targets.
• performance optimizationPerformance optimization is the systematic modification of software or its operating conditions to improve measured execution speed, responsiveness, throughput, or resource efficiency while preserving required behavior.
• CPythonCPython is the reference implementation of the Python programming language, written primarily in C and providing its interpreter and runtime.
• MapReduceMapReduce is a computational pattern that applies map operations to input elements or partitions and combines the intermediate results through reduction or aggregation.
• tensor shapeA tensor shape is an ordered sequence of dimension sizes specifying the extent of a tensor or array along each axis.
• multithreadingMultithreading is a concurrency technique in which multiple threads of execution operate within a program or process, with their work interleaved or running in parallel.
• atomic typeAn atomic type is a type whose supported accesses and operations are indivisible with respect to concurrent execution and provide defined synchronization semantics.
• shared memoryShared memory is a memory region accessible by multiple execution contexts for accessing or exchanging data.
• pandaspandas is an open-source Python library that provides labeled data structures and tools for manipulating and analyzing tabular and other structured data.
• MapA map is an associative data structure that stores key-value pairs and supports retrieving values by key.
• vectorizationVectorization is representing data or computations in vector or array form so they can be processed collectively rather than as separate scalar values.
• compressionCompression is the transformation of information into a more compact representation by encoding redundancy or omitting detail while retaining information needed for recovery or use.
• CPU cacheA CPU cache is small, fast memory that stores copies of data or instructions near a processor for quicker access.
• multiprocessingMultiprocessing is a programming technique that executes work in multiple operating-system processes, enabling concurrent or parallel execution across available processor resources.
• process poolA process pool is a managed collection of worker processes that executes submitted tasks concurrently.
• ZarrZarr is a storage format and software ecosystem for chunked, compressed, multidimensional arrays organized in hierarchical groups.
• schedulerA scheduler is a system component that selects, orders, dispatches, or assigns execution resources to tasks according to timing, dependency, placement, or other scheduling criteria.
• hardwareHardware is the physical equipment of a computing system and the organization and execution characteristics of its processors and other components.
• setA set is a collection of unique elements in which membership is determined by whether an element is present, not by duplicate occurrences.
• CuPyCuPy is an open-source Python library providing a NumPy-compatible interface for GPU-accelerated numerical computing.
• GPUA graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and data-parallel numerical workloads.
• asynchronous programmingAsynchronous programming structures operations so they can proceed or complete without requiring the initiating execution flow to wait for them.
• broadcastingBroadcasting is the distribution or implicit replication of data from one source across multiple recipients or compatible positions for communication or computation.
• missing valueA missing value is a representation or condition indicating that an expected data value or object reference is absent, unknown, unavailable, or inapplicable.
• BloscBlosc is a high-performance lossless compression library optimized for binary data and designed to exploit blocking and modern processor capabilities.
• ParquetApache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
• line_profilerline_profiler is a Python profiling tool that measures execution time for individual source-code lines.
• CPUA CPU, or central processing unit, is the hardware processor that executes computer instructions and performs general-purpose computation.
• array programmingArray programming is a programming technique that expresses operations over whole arrays rather than iterating imperatively over individual elements.
• dask.distributeddask.distributed is a Dask package that provides a task scheduler, workers, clients, and distributed execution infrastructure.
• memoryMemory is a storage resource or mechanism that retains data or computational state for access during computation or later use.
• memoryviewA memoryview is a Python object that exposes a view of a memory buffer containing homogeneous data without copying it.
• strideA stride is the step interval between successive positions or elements accessed, selected, or processed along a dimension or sequence.
• np.shares_memorynp.shares_memory is a NumPy function that determines whether two arrays share memory.
• NumExprNumExpr is a Python library that evaluates numerical array expressions using optimized operations and reduced intermediate memory.
• Moore's lawMoore's law is the empirical observation that transistor density on integrated circuits historically doubled approximately every two years, increasing computational capability.
• big dataBig data is data whose volume, velocity, variety, or complexity challenges conventional storage, processing, or analysis methods.
• GPU kernelA GPU kernel is a program or function executed on a graphics processing unit to perform a computation, often in parallel across many threads.
• fsspecfsspec is a Python library that provides a uniform interface for accessing filesystems and file-like objects across local and remote storage systems.
• model architectureModel architecture is the structural organization of a computational system’s components, relationships, and operations, defining how it processes information or carries out behavior.
• thread blockA thread block is a group of GPU threads scheduled as a unit that can cooperate through shared memory and synchronization.
• arrayAn array is an indexed, ordered collection of elements arranged in one or more dimensions, with elements accessed by position.
• CUDACUDA is NVIDIA’s parallel computing platform and programming model for executing general-purpose computations on NVIDIA GPUs.
• columnar dataColumnar data is tabular data organized and stored by column rather than by complete row, enabling efficient access to selected fields and data compression.
• data localityData locality describes the relationship between where data is located and where or how it is accessed or processed, including proximity, reuse, and location constraints.
• memory allocationMemory allocation is the process of reserving or assigning storage for program data and objects during execution.
• object overheadObject overhead is the memory required to represent an object beyond the memory occupied by its payload or explicitly stored data.
• executorAn executor is a software component or interface that accepts tasks or actions and arranges for their execution by an appropriate runtime or worker.
• cuda.jitcuda.jit is a Numba decorator that compiles a Python function for execution as CUDA device code or a GPU kernel.
• concurrencyConcurrency is the structuring or execution of multiple computational activities so their progress overlaps in time, whether or not they run simultaneously.
• Zarr arrayA Zarr array is a persistent multidimensional array whose data and metadata are stored in independently addressable chunks.
• Edholm's lawAn empirical observation that telecommunications data rates tend to double approximately every 18 months.
• exponential growth curveAn exponential growth curve represents a quantity that increases by a constant multiplicative factor over equal increments of its input.
• unstructured dataUnstructured data is information that lacks a predefined schema or consistently organized fields.
• memory boundA memory-bound workload is one whose performance is limited primarily by the rate of accessing or transferring data rather than by computational throughput.
• basebase is a NumPy array attribute referencing the first array object in the chain from which a view derives.
• in-memory processingIn-memory processing performs computations primarily on data held in main memory rather than repeatedly accessing persistent storage.
• QuadLifeQuadLife is a colored extension of Conway's Game of Life in which live cells can have four color states and additional color-transition rules.
• local networkA local network is a computer network connecting devices within a limited physical or administrative area, such as a home, office, or campus.
• bounds checkingBounds checking verifies that an index or access lies within the valid limits of a data structure before the access is performed.
• row iterationRow iteration is the technique of processing a tabular dataset one record at a time through sequential procedural evaluation.
• asynchronous serverAn asynchronous server is a server whose operations can yield while waiting for input or other external events so that other client work can proceed.
• ufuncA ufunc is a NumPy universal function that performs elementwise operations on arrays and supports broadcasting.
• chunk granularityChunk granularity is the amount of data or content assigned to each unit when a larger source is divided into chunks for processing, storage, transfer, or retrieval.
• loopA loop is a programming control-flow construct that repeatedly executes a block of instructions according to an iteration rule or continuation condition.
• cloud computingCloud computing is the on-demand delivery of computing resources and services over a network from remotely managed infrastructure.
• transistor densityTransistor density is the number of transistors integrated per unit area of a semiconductor device.
• repartitionRepartitioning is the redistribution of records or items across partitions according to a partitioning rule, potentially changing the number of partitions.
• Plasma serverPlasma is an in-memory object store and server associated with Apache Arrow that enables processes to exchange objects efficiently.
• interprocess communicationInterprocess communication is the exchange of data, information, or coordination signals between independently executing processes.
• sequential processingSequential processing is the execution of operations or tasks one after another in a defined order rather than concurrently.
• persistpersist is a Dask collection operation that starts computation and retains the resulting partitions for reuse in subsequent computations.
• Game of LifeConway's Game of Life is a cellular automaton whose grid cells change state across discrete generations according to local neighborhood rules.
• fancy indexingFancy indexing is selecting multiple array elements or positions with a collection of integer, Boolean, or label-based selectors.
• callbackA callback is a function or other callable supplied to software for invocation later in response to a specified event, condition, or point in execution.
• picklePickle is Python’s standard-library facility for serializing Python objects to byte streams and reconstructing them from those streams.
• transpositionTransposition is an operation that exchanges a matrix’s rows and columns or, more generally, reorders the axes of a tensor without changing its values.
• BLASBLAS is a standard interface and collection of low-level routines for efficient vector and matrix operations in numerical computing.
• Intel MKLIntel MKL, now Intel oneMKL, is a library of optimized mathematical and scientific computing routines, including implementations of BLAS and LAPACK.
• OpenBLASOpenBLAS is an open-source optimized implementation of the BLAS and LAPACK numerical linear algebra interfaces.
• execution modelAn execution model is an abstract set of rules governing how a computational system runs, schedules, coordinates, orders, and triggers computations.
• sys.getsizeofsys.getsizeof is a Python standard-library function that returns an object's shallow memory size in bytes.
• contiguous memory layoutA contiguous memory layout stores logically adjacent data elements at adjacent memory addresses, without gaps between them.
• cuda.gridcuda.grid is a Numba CUDA intrinsic that computes a thread's global index in a one-, two-, or three-dimensional execution grid.
• cdefcdef is a Cython declaration for defining C-level functions, variables, or types without automatically creating a Python-callable interface.
• larger-than-memory dataLarger-than-memory data is data whose total size exceeds the available main memory for processing it all at once.
• high-performance computingHigh-performance computing is the use of powerful computing systems and parallel, distributed, or specialized techniques to execute computationally intensive workloads efficiently.
• UDPUDP (User Datagram Protocol) is a connectionless transport-layer protocol that sends datagrams without guaranteeing delivery, ordering, or duplicate suppression.
• horizontal scalingHorizontal scaling is adjusting system capacity by adding or removing computing nodes or equivalent instances that share or distribute the workload.
• row groupA row group is a storage and processing unit containing a subset of rows in a columnar data file.
• generatorA generator is a language construct or computation that produces values incrementally on demand, typically suspending and resuming while preserving its state.
• Dask dataframeA Dask dataframe is a partitioned tabular data structure whose operations are represented as a Dask task graph and whose partitions are typically pandas data frames.
• Python wrapperA Python wrapper is a Python interface that exposes the operations of an underlying library, service, or native implementation.
• bisection algorithmA bisection algorithm repeatedly divides an ordered search range to locate a target or insertion position.
• object caching and reuseObject caching and reuse is a memory-management technique that retains existing objects for repeated use instead of repeatedly creating equivalent objects.
• memory hierarchyA memory hierarchy is an arrangement of storage levels with differing capacities, access speeds, latencies, costs, or persistence.
• concurrent.futuresconcurrent.futures is a Python standard-library module providing high-level interfaces for asynchronously executing and coordinating callable tasks with threads or processes.
• time-space tradeoffA time-space tradeoff is a design choice that exchanges computational time for storage or memory, or storage or memory for computational time.
• wrap-aroundWrap-around is a boundary-indexing technique that connects opposite edges of a finite grid so indexing past one edge continues at the other edge.
• nogilnogil is a Cython function qualifier indicating that a function can execute without holding CPython's global interpreter lock.
• bit arrayA bit array is a compact, indexed data structure whose elements are individually accessible binary values.
• instrumentationInstrumentation is the addition or activation of mechanisms in software or systems to observe, measure, or record their behavior during execution.
• applyIn pandas, apply is a method that applies a callable across Series values or along a DataFrame axis and returns the results.
• iterrowsiterrows is a pandas DataFrame method that yields row index and row data pairs for sequential row traversal.
• itertuplesitertuples is a pandas DataFrame method that yields each row as a tuple, optionally including the row index.
• SeriesA Series is pandas’ one-dimensional labeled data structure that associates values with an index.
• ZstandardZstandard, or ZSTD, is a general-purpose lossless compression algorithm and format with configurable trade-offs between compression ratio and speed.
• benchmarkA benchmark is a defined task, dataset, workload, procedure, or reference measure used to assess or compare performance or capability under specified criteria or conditions.
• cProfilecProfile is a Python standard-library deterministic profiler that records function-call counts and execution-time statistics.
• type inferenceType inference is the automatic deduction of program or expression types from available values, context, and constraints without requiring explicit type annotations.
• eager evaluationEager evaluation is an execution strategy that computes expressions or performs operations immediately rather than deferring them until their results are needed.
• C intC int is a signed integer type whose range and representation are determined by the C implementation.
• unsigned integerAn unsigned integer is an integer represented using a range of nonnegative whole-number values, with the range determined by its representation.
• data conversionData conversion is the transformation of a value or data from one type, format, or representation to another, explicitly or implicitly.
• mutationMutation is an alteration to an existing value, state, data structure, or representation.
• numerical overflowNumerical overflow is the condition in which an arithmetic result falls outside the range representable by its numeric type or format.
• non-object arrayA non-object array is an array whose elements are stored as raw values of a specific data type rather than as references to general objects.
• data transformationA data transformation is an operation or function that maps input data or its representation to modified data or a different representation.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.