Columnar data
Columnar data is tabular data organized and stored by column rather than by complete row, enabling efficient access to selected fields and data compression.
Cloud Observability in Action
Columnar storage is an arrangement of tabular data that stores values from the same column together for efficient analytical access.
- How it is used
- Supports analytical observability workloads involving high-cardinality telemetry and large historical datasets.
- How deep it goes
- Explains physical layout, OLTP-versus-OLAP trade-offs, telemetry setup, querying, and profiling storage.
- Examples
- Uses row-versus-column diagrams, a ClickHouse Docker Compose deployment, OpenTelemetry configuration, SQL queries, and persistent profile storage.
- What you can do after
- Evaluate storage layouts and select columnar backends for high-cardinality analytical workloads while weighing operational costs.
Read how Cloud Observability in Action explains columnar data →
Fast Python: High performance techniques for large datasets
- How it is used
- Optimizes tabular analytics by reducing memory use, accelerating I/O and computation, and enabling efficient data exchange.
- How deep it goes
- Explains layout, typing, encoding, compression, and interoperability, but defers detailed persistent formats and on-disk implementations.
- Examples
- Uses pandas dtype optimization, PyArrow CSV conversion, Parquet storage, and New York City taxi records.
- What you can do after
- Operate/configure pandas, Arrow, and Parquet workflows to select, type, compress, store, and transfer tabular columns.
Read how Fast Python: High performance techniques for large datasets explains columnar data →
Just Use Postgres!
A columnar store is a storage layout that organizes values by column to improve analytical scans over selected columns and large data ranges.
- How it is used
- Optimizes analytical scans over large or historical time intervals when indexed row-store access becomes less efficient.
- How deep it goes
- Defines partition-level columnar storage and its placement, but defers Hypercore implementation details to official documentation.
- Examples
- Uses a policy that retains three recent months in indexed row storage and moves older partitions into an unindexed columnar store.
- What you can do after
- Operate/configure TimescaleDB by separating recent indexed row data from older columnar partitions.
Architecting an Apache Iceberg Lakehouse
A columnar format stores values from the same field together to support efficient analytical reading and compression.
- How it is used
- Supports analytical scans, aggregations, dashboards, and long-range queries without reading unused fields from transactional rows.
- How deep it goes
- Explains storage layout, column pruning, compression, vectorized execution, and the division between Parquet and Iceberg.
- Examples
- Uses Parquet files, Iceberg tables, and conversion from CSV, JSON, Avro, or row-based data as concrete artifacts.
- What you can do after
- Evaluate storage formats and plan columnar conversion for analytical pipelines and Iceberg migrations.
Read how Architecting an Apache Iceberg Lakehouse explains columnar data →
Related concepts
- Parquetincludes · Apache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
- data lakeused by · A data lake is a repository that stores large volumes of data in raw or lightly processed form and varied formats for later processing and analysis.
- analytical processinguses · Interactive processing of data through complex queries, transformations, and aggregations to support analysis, reporting, or decision-making.
- Apache Arrowincludes · Apache Arrow is an open, language-independent specification and software ecosystem for representing and processing columnar data in memory.
- ClickHouseincludes · ClickHouse is an open-source, column-oriented database management system designed for high-performance analytical queries.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.