How deeply does Architecting an Apache Iceberg Lakehouse cover Redpanda?

Explains configuration, ingestion modes, catalogs, schemas, partitioning, failures, and operational tradeoffs without detailing internals.

Redpanda

As explained in Architecting an Apache Iceberg Lakehouse

Redpanda is a Kafka-compatible distributed event-streaming platform designed for low-overhead, high-performance message processing.

Redpanda is a Kafka-compatible streaming platform that materializes topics directly as Apache Iceberg tables in cloud object storage, avoiding intermediate ETL pipelines. Downstream systems can query the resulting Parquet-backed tables through engines such as Spark, Flink, Snowflake, and ClickHouse. section 6.4.8

A practical Redpanda-to-Iceberg design includes:

Enable iceberg_enabled and configure each topic with redpanda.iceberg.mode: key_value preserves binary payloads, while value_schema_id_prefix and value_schema_latest support schema-based structured data. section 6.4.8
Choose a REST catalog for shared production metadata or a filesystem catalog for object-storage metadata that requires manual client refreshes. section 6.4.8
Use topic-level partitioning to reduce scanned files, and monitor CPU usage, file sizes, commit frequency, and Iceberg metadata growth. section 6.4.8 section 6.5.1

Avro and Protobuf schema evolution supports additions and column reordering. Conversion failures can be routed to a dead-letter queue table for inspection and reprocessing; JSON remains possible through key_value mode. section 6.4.8

How Architecting an Apache Iceberg Lakehouse covers it

Cover of Architecting an Apache Iceberg Lakehouse

liveBook manning.com

Read Architecting an Apache Iceberg Lakehouse on liveBook Get the book at manning.com

Related concepts

Apache Kafkaalternative to · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
Apache Iceberguses · Apache Iceberg is an open table format for managing large analytical datasets with metadata, schema evolution, snapshots, and transactional operations across distributed or object storage.
streamingused by · Streaming is the incremental handling of a sequence of data items, including their processing, transfer, or consumption as they become available.
Strimzialternative to · Strimzi is an open-source Kubernetes operator and set of resources for deploying and managing Apache Kafka clusters and related components.
Parquetproduces · Apache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
dev serviceused by · Dev Services is a Quarkus feature that automatically starts and manages disposable supporting service instances during development and testing.

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.