liveBook · Manning

Physical data type

As explained in DuckDB in Action

A physical type is the low-level representation used to store a value on disk in a Parquet file.

A physical data type is the low-level representation used to store a value on disk in a Parquet file. Parquet supports a small set of physical types, including Boolean, integers of various sizes, and byte arrays. Logical types add metadata that tells readers how to interpret those stored values: a date, for example, can be stored as a numeric INT32 value while being exposed as DATE. section 5.4 section 5.5

Use physical types deliberately when converting and inspecting data:

  • Convert source values to precise logical types before writing Parquet; parsing ranking_date with %Y%m%d produces a DATE rather than leaving it as an integer. section 5.4
  • Inspect the user-facing schema with DESCRIBE, then inspect the on-disk representation with parquet_schema. section 5.5
  • Choose the smallest numeric physical type that can hold the data when performance matters, because smaller numerical values support more parallel vector operations. section 5.5

Dates stored as integers

A Parquet schema can expose ranking_date as DATE while storing it physically as INT32 with DATE metadata. Text columns such as name_first use BYTE_ARRAY with UTF8 interpretation, while rank and points use INT64. 

┌───────────────┬────────────┬────────────────┬──────────────┐
│     name      │    type    │ converted_type │ logical_type │
│    varchar    │  varchar   │    varchar     │   varchar    │
├───────────────┼────────────┼────────────────┼──────────────┤
│ duckdb_schema │            │                │              │
│ ranking_date  │ INT32      │ DATE           │              │
│ rank          │ INT64      │ INT_64         │              │
│ points        │ INT64      │ INT_64         │              │
│ player_id     │ INT64      │ INT_64         │              │
│ name_first    │ BYTE_ARRAY │ UTF8           │              │
│ name_last     │ BYTE_ARRAY │ UTF8           │              │
│ hand          │ BYTE_ARRAY │ UTF8           │              │
│ dob           │ INT32      │ DATE           │              │
│ ioc           │ BYTE_ARRAY │ UTF8           │              │
│ height        │ INT64      │ INT_64         │              │
├───────────────┴────────────┴────────────────┴──────────────┤
│ 11 rows                                          4 columns │
└────────────────────────────────────────────────────────────┘

Inspect the relationship between each column's physical type and its converted or logical interpretation. section 5.5

How DuckDB in Action covers it

How deep it goes
Explains physical storage, logical annotations, schema inspection, and numeric-width trade-offs, but stops short of encoding internals.
Examples
Demonstrates INT32 dates, INT64 numeric columns, and BYTE_ARRAY UTF8 strings in an ATP rankings Parquet file.
What it assumes
Assumes basic SQL and column types while teaching Parquet storage and logical interpretation through CSV conversion.
What you can do after
Diagnose Parquet schemas with DuckDB, verify logical coercions, and select narrower numeric representations when values permit.
How it connects
Connects Parquet physical types with logical types, schemas, CSV inference, DuckDB, and vectorized query performance.
Cover of DuckDB in Action
liveBookmanning.com
Read DuckDB in Action on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of DuckDB in Action. The section links point to the parts of the book it draws on.