Apache Arrow
Apache Arrow is an open, language-independent specification and software ecosystem for representing and processing columnar data in memory.
Fast Python: High performance techniques for large datasets
- How it is used
- Uses Arrow to complement pandas with faster loading, computation, interoperability, and Parquet writing.
- How deep it goes
- Explains column types, missing-value representation, conversion costs, and shared-memory workflows while limiting Arrow analytics to a brief treatment.
- Examples
- Benchmarks taxi CSV loading and tip calculations, then uses taxi data for shared-memory processing and chunked Parquet conversion.
- What you can do after
- Apply PyArrow to load and analyze tabular data, convert it to pandas, and write Parquet files.
Read how Fast Python: High performance techniques for large datasets explains Apache Arrow →
Architecting an Apache Iceberg Lakehouse
Apache Arrow is a software project and standardized in-memory columnar data format for efficient data interchange across systems and programming languages.
- How it is used
- Provides the columnar interchange layer for high-performance federated analytics and cross-language connectivity.
- How deep it goes
- Explains columnar representation, zero-copy exchange, vectorized execution, and related protocols, but not the format specification itself.
- Examples
- Uses Dremio query execution, BI tools, data science notebooks, Arrow Flight, and transparent reflections as concrete applications.
- What you can do after
- Evaluate/understand how Arrow reduces analytical data-transfer overhead and supports columnar execution in federated platforms.
Read how Architecting an Apache Iceberg Lakehouse explains Apache Arrow →
DuckDB in Action
Apache Arrow is a language-independent specification and software ecosystem for representing columnar data in memory.
- How it is used
- Connects DuckDB, Python, Polars, and other analytical tools through shared in-memory columnar data.
- How deep it goes
- Defines the memory model and demonstrates conversions, filtering, and slicing without detailing Arrow internals.
- Examples
- Uses DuckDB relations, Polars DataFrames, PyArrow filtering, typed columns, and population records.
- What you can do after
- Apply it via DuckDB, Polars, or PyArrow to exchange, filter, and inspect columnar data in Python workflows.
Also covered in
- Pandas Workout: 200 exercises to make you a stronger data analyst
- Cloud Observability in Action
- Time Series Forecasting Using Foundation Models
What to know first
- CSVCSV is a text format for tabular data in which records contain fields separated by commas, with conventions for quoting fields that contain special characters.
- data frameA data frame is a two-dimensional tabular data structure organized into labeled rows and columns, often with columns that can contain different types of values.
- PythonPython is a high-level, general-purpose, dynamically typed programming language with readable syntax and support for multiple programming paradigms.
- structured dataStructured data is information organized according to a defined schema or arrangement that makes its elements and relationships consistently interpretable and processable.
Related concepts
- PyArrowused by · PyArrow is a Python library providing access to Apache Arrow columnar data structures, memory formats, and related tools.
- columnar storagekind of · Columnar storage is a data layout that stores values grouped by column rather than by row.
- datasetused by · A dataset is an organized collection of related data items, records, observations, or examples treated as a unit for storage, processing, or analysis.
- Dremio Cloudused by · Dremio Cloud is a managed cloud data lakehouse platform for querying, managing, and accelerating access to data across diverse sources.
- fsspecuses · fsspec is a Python library that provides a uniform interface for accessing filesystems and file-like objects across local and remote storage systems.
Questions about Apache Arrow
What is Apache Arrow?
Apache Arrow is an open, language-independent specification and software ecosystem for representing and processing columnar data in memory.
Which Manning books explain Apache Arrow?
Fast Python: High performance techniques for large datasets, Architecting an Apache Iceberg Lakehouse and DuckDB in Action.
What should I know before learning Apache Arrow?
Manning books that teach Apache Arrow usually introduce CSV, data frame, Python and structured data first.
What is Apache Arrow a kind of?
Apache Arrow is a kind of columnar storage. Columnar storage is a data layout that stores values grouped by column rather than by row.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


