Apache Arrow
Apache Arrow is an open, language-independent specification and software ecosystem for representing and processing columnar data in memory.
Fast Python: High performance techniques for large datasets
- How it is used
- Uses Arrow to accelerate pandas data loading and enable columnar-data exchange across processes and languages.
- How deep it goes
- Works through CSV loading, types, compute and shared-memory examples, while stopping short of a full treatment of Arrow analytics.
- Examples
- Benchmarks a taxi CSV load, computes a mean tip fraction, and shares taxi data between local processes.
- What you can do after
- Apply it via PyArrow to inspect CSV memory use, convert tables for pandas, and write Parquet.
Read how Fast Python: High performance techniques for large datasets explains Apache Arrow →
Architecting an Apache Iceberg Lakehouse
Apache Arrow is a software project and standardized in-memory columnar data format for efficient data interchange across systems and programming languages.
- How it is used
- Provides the columnar interchange layer for high-performance federated analytics and cross-language connectivity.
- How deep it goes
- Explains columnar representation, zero-copy exchange, vectorized execution, and related protocols, but not the format specification itself.
- Examples
- Uses Dremio query execution, BI tools, data science notebooks, Arrow Flight, and transparent reflections as concrete applications.
- What you can do after
- Evaluate/understand how Arrow reduces analytical data-transfer overhead and supports columnar execution in federated platforms.
Read how Architecting an Apache Iceberg Lakehouse explains Apache Arrow →
DuckDB in Action
Apache Arrow is a language-independent specification and software ecosystem for representing columnar data in memory.
- How it is used
- Connects DuckDB, Python, Polars, and other analytical tools through shared in-memory columnar data.
- How deep it goes
- Defines the memory model and demonstrates conversions, filtering, and slicing without detailing Arrow internals.
- Examples
- Uses DuckDB relations, Polars DataFrames, PyArrow filtering, typed columns, and population records.
- What you can do after
- Apply it via DuckDB, Polars, or PyArrow to exchange, filter, and inspect columnar data in Python workflows.
Related concepts
- PyArrowused by · PyArrow is a Python library providing access to Apache Arrow columnar data structures, memory formats, and related tools.
- Parquetused by · Apache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
- fsspecuses · fsspec is a Python library that provides a uniform interface for accessing filesystems and file-like objects across local and remote storage systems.
- datasetused by · A dataset is an organized collection of related data items, records, or observations treated as a unit for storage, processing, analysis, or computation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.