Columnar data stores values from the same field together instead of storing complete records row by row. This suits analytical queries that scan many rows but use only a few columns, avoiding unnecessary reads from transactional storage. section 1.2 section 2.1
Columnar layouts provide several practical advantages:
- Read only the columns required by a query, reducing I/O during scans and aggregations. section 2.1
- Compress values efficiently because each stored group contains the same field and data type. section 2.1
- Process batches of values through vectorized execution, improving cache locality and SIMD use. section 2.1
Parquet in a lakehouse
Apache Parquet provides the physical columnar representation, while Apache Iceberg organizes files and manages schemas, snapshots, statistics, and table semantics. Converting CSV, JSON, or other row-based inputs into Parquet can support an Iceberg migration, but may require staging or conversion pipelines. section 2.1 section 4.1.3
