Columnar data organizes tabular values by column rather than complete row. This layout supports selective field access, typed representation, compression, and efficient transfer between processes or languages. section 7.0 section 8.2
Use columnar representations when performance depends on memory, I/O, or selecting only part of a table:
- Read only the columns required by an analysis instead of loading complete rows. section 8.2
- Assign narrow, appropriate types because overly general column types increase memory use and slow operations. section 7.1.2
- Apply compression and encodings such as dictionaries or run-length encoding when columns contain repeated patterns. section 8.5
- Use Apache Arrow when columnar data must move across languages, processes, or machines. section 7.0
Converting taxi data to Parquet
Apache Arrow can read a compressed CSV table and write it as Parquet: `table = csv.read_csv("yellow_tripdata_2020-01.csv.gz")`; `pq.write_table(table, "202001.parquet")`. Parquet stores typed data by column, enabling selective access and compression for the New York City taxi dataset. section 8.2 section 8.5
Use the architecture diagram to trace how Python wrappers connect pandas-oriented workflows with Arrow's lower-level implementations, persistence layers, and interprocess or intermachine communication. section 7.4.1
