Chapter 3: Hands-on with Apache Iceberg

Introduction

Introduces a hands-on chapter that will build a laptop-based Iceberg lakehouse and use it to create tables, run queries, and visualize results.

3.1 Our example

Introduces a simplified fashion-retail example and traces its sales data from PostgreSQL through Iceberg tables to BI dashboards.

• Sales data lakehouse workflow: Trace the example’s sales data from its PostgreSQL source through Spark-written Iceberg tables and Dremio views to Superset dashboards.
  Why it matters: This gives you a map of how the hands-on example turns operational data into business-facing analysis.
• Simplified example design choices: Explain why the example uses familiar sales data and a deliberately simple setup rather than exploring every possible ingestion architecture.
  Why it matters: The simplified setup helps you separate the example’s teaching purpose from the choices a production architecture might require.

Key terms: PostgreSQL, Apache Spark, Apache Iceberg tables, Dremio, virtual views, Apache Superset dashboards, Nessie, ingestion architectures

3.2 Setting up an Apache Iceberg environment

Introduces the services and their roles in a local, end-to-end Iceberg lakehouse example.

• Local lakehouse service roles: Describe the services in the Docker Compose example and the role each plays, distinguishing PostgreSQL as the data source from the lakehouse components.
  Why it matters: These roles provide a map for following data from the source through the local analytics stack.

Key terms: PostgreSQL, MinIO, Apache Spark, Nessie, Dremio, Apache Superset

3.2.1 Prerequisite: Installing Docker

This subsection establishes Docker Desktop as the prerequisite for running the hands-on services in containers and checking that it is installed.

• Docker prerequisite and containers: Explain why the hands-on environment requires Docker Desktop and describe how to check the installation after installing it.
  Why it matters: Docker lets you use the example services without installing each one directly on your computer.

Key terms: Docker Desktop, containers, Nessie, Apache Spark, Dremio

3.2.2 Creating the Docker Compose file

Defines a local Docker Compose lakehouse and explains how its catalog, storage, source database, processing engine, query engine, and BI tool connect.

• Shared local service environment: Explain how the Docker Compose file brings the lakehouse services together locally, using a shared network for container communication and mapped ports for access from the host.
  Why it matters: This is the connection between the individual service definitions and a working local lakehouse.
• Catalog and object storage: Describe what Nessie and MinIO provide in this setup, including why the specified Nessie backend is suited to development rather than persistent use.
  Why it matters: The catalog and object store serve different roles, and the catalog’s in-memory setting limits this setup to development or testing.
• Source database and Spark: Trace the roles of PostgreSQL and Spark in the example, from an operational data source to processing that can read and write S3-compatible storage.
  Why it matters: These services supply the example’s source data and a way to process it into the lakehouse.
• Query and dashboard path: Explain how Dremio and Superset serve downstream analytics in this environment: Dremio provides SQL queries over Iceberg tables, and Superset uses it for visualizations.
  Why it matters: It explains how data in the lakehouse becomes available for SQL exploration and dashboards.

Key terms: docker-compose.yml, lakehouse-net, port mapping, Project Nessie, in-memory storage backend, REST API, MinIO, S3-compatible API, PostgreSQL, operational data source

3.2.3 Running the environment

This section starts and verifies the Docker Compose lakehouse, then explains how its containers work together to ingest data and serve dashboards.

• Environment startup and verification: Describe how to start the environment from the directory containing docker-compose.yml and check that its containers are running. Explain that the lakehouse is ready when all the containers are running.
  Why it matters: The later exercises depend on the services being up before you use them.
• Spark ingestion and table metadata: Trace how a Python-submitted Spark job reads from PostgreSQL and writes into the Iceberg lakehouse. Explain the roles of Nessie, MinIO, and the catalog update in that flow.
  Why it matters: This connects the source database, processing job, file storage, and table catalog into one ingestion path.
• Dremio and dashboard access: Explain how Dremio and a BI tool use the ingested Iceberg data. Describe Dremio’s shared Nessie and MinIO configuration and Superset’s connection to Dremio.
  Why it matters: It shows how ingested tables become datasets that people can explore and visualize.

Key terms: Docker Compose, docker-compose.yml, running containers, Python script, Spark job, Postgres, Nessie, MinIO, catalog, Dremio

3.2.4 Accessing services

This section gives the local addresses and credentials for checking that the five lakehouse services are accessible.

• Local service access checks: Describe how to access each running service to check that it is working, distinguishing the browser-based services from PostgreSQL client access.
  Why it matters: Checking each service confirms that the local environment is accessible before you use it.

Key terms: MinIO, Dremio, Spark, Jupyter Notebook, PostgreSQL, Superset

3.3 Creating Iceberg tables in Spark

Introduces the retail-sales data flow from PostgreSQL into Iceberg and explains how MinIO and Nessie support storage and shared access.

• Transactional Data to Analytics: Describe how the example moves retail sales data from an operational PostgreSQL source into Iceberg tables for downstream analytics.
  Why it matters: This connects the hands-on exercise to the broader purpose of an Iceberg lakehouse.
• Storage and Catalog Roles: Distinguish MinIO’s role as the object-storage backend from Nessie’s role as the catalog that tracks and governs Iceberg tables for other tools to discover.
  Why it matters: Separating storage from catalog responsibilities explains how multiple tools can work with the same tables.

Key terms: PostgreSQL, Apache Spark, Apache Iceberg tables, transactional sources, MinIO, object-storage backend, Nessie Iceberg catalog

3.3.1 Populating the PostgreSQL database

This section prepares mock sales data in PostgreSQL as the source for later ingestion into Iceberg.

• PostgreSQL sample data preparation: Describe how the section prepares PostgreSQL as a source for Iceberg ingestion: enter the Postgres shell, create and populate a fashion_sales_data table with mock sales data, then exit.
  Why it matters: The Iceberg ingestion exercise needs source data to load.

Key terms: PostgreSQL, Postgres shell, fashion_sales_data, Iceberg tables

3.3.2 Starting Apache Spark

This section prepares Spark’s notebook and MinIO storage, then shows how to find a reliable network address for connecting them.

• Spark workspace and storage: Describe how to prepare the hands-on environment: create a MinIO bucket named warehouse for Iceberg table data, then open the running Spark environment in Jupyter and create a Python notebook for PySpark code.
  Why it matters: These steps give the exercise both a place to run Spark code and a destination for its Iceberg data.
• MinIO container network address: Explain how to identify the MinIO container’s IPv4 address on the correct Docker network and why the exercise uses that address in the PySpark script instead of relying on the container name.
  Why it matters: Finding the container IP provides a connection option when its name does not resolve in this exercise.

Key terms: MinIO, warehouse, Jupyter Notebook, PySpark, Docker network, IPv4 address, MinIO container, Docker DNS

3.3.3 Configuring Apache Spark for Iceberg

This section configures a PySpark session to use Nessie as the Iceberg catalog and MinIO as its storage backend, then starts the session.

• Catalog and storage variables: Explain what CATALOG_URI, WAREHOUSE, and STORAGE_URI tell Spark about the Nessie catalog and MinIO storage.
  Why it matters: These addresses connect Spark’s table metadata to the location of the actual data.
• Spark catalog integration settings: Describe how the Spark configuration enables Iceberg operations with Nessie and MinIO, including the required libraries and catalog and storage settings.
  Why it matters: The configuration lets one Spark job use Nessie-managed Iceberg tables whose data is stored in MinIO.
• Configured Spark session startup: Explain how the configured Spark session is started and what the startup message confirms.
  Why it matters: The session is the entry point for the subsequent Spark reads and writes.

Key terms: CATALOG_URI, WAREHOUSE, STORAGE_URI, Nessie, MinIO, SparkConf, spark.jars.packages, Iceberg SparkCatalog, S3FileIO, SparkSession.builder.config(conf=conf).getOrCreate()

3.3.4 Loading data from PostgreSQL into Iceberg

This section shows how to load PostgreSQL sales data into an Iceberg table, with data files in MinIO and table metadata registered in Nessie.

• Iceberg catalog namespaces: Explain what a namespace is and why the example creates one before writing the table.
  Why it matters: It helps distinguish how tables are organized in a catalog from where their files are stored.
• PostgreSQL to Iceberg load: Trace how the example moves sales data from PostgreSQL into Iceberg and distinguish the roles of MinIO and Nessie.
  Why it matters: The load illustrates how the storage and catalog services work together to make an Iceberg table usable.

Key terms: namespace, Iceberg catalogs, Nessie, sales_data, Spark DataFrame, Iceberg table, MinIO

3.3.5 Verifying data storage in MinIO

Shows how to verify in MinIO that Spark stored the ingested Iceberg table and how to identify its data and metadata files.

• MinIO table storage verification: Describe how to find the sales_data Iceberg table in MinIO and explain what the files there show about the completed ingestion.
  Why it matters: Checking the object store confirms that the ingestion produced both table data and the metadata Iceberg uses to describe it.

Key terms: MinIO, warehouse bucket, sales/sales_data_UUID/, Parquet, JSON, Avro

3.4 Reading Iceberg tables with Dremio

Introduces querying existing Iceberg tables with Dremio and explains how Iceberg metadata can make analytics on the data lake faster without duplicating data.

• Analytics without data duplication: Describe Dremio as a lakehouse query engine that can query Iceberg tables on the data lake using standard SQL, giving a warehouse-like experience without moving or replicating the data.
  Why it matters: It connects the hands-on query experience to the lakehouse benefit of analyzing data without making another copy.
• Metadata-based query pruning: Explain that Iceberg’s partition information and file-level statistics let Dremio skip unnecessary data during queries, improving speed as the dataset scales.
  Why it matters: It explains why querying data directly on the lake can still be fast.

Key terms: Dremio, Apache Iceberg tables, standard SQL, data lake, metadata pruning, partition information, file-level statistics

3.4.1 Starting Dremio

This subsection shows how to open the running Dremio UI and explains what Dremio can do with data sources and Iceberg tables.

• Dremio UI access: Describe how to access Dremio in the Docker Compose environment: open http://localhost:9047 in a browser and log in with the admin account created earlier.
  Why it matters: Opening the UI is the starting point for working with Dremio in this example.
• Dremio data access capabilities: Explain that Dremio can connect to data sources and lakehouse catalogs such as Nessie, run SQL on datasets, and query across sources; the catalog connection lets it find and work with tracked Iceberg tables.
  Why it matters: These capabilities explain why the example uses Dremio to access Iceberg tables.

Key terms: Docker Compose, Dremio UI, admin account, data sources, SQL, federated queries, Nessie, lakehouse catalogs, Iceberg tables

3.4.2 Connecting Dremio to the Nessie catalog

Shows how to add Nessie as a Dremio source so Dremio can find and query Iceberg tables, using MinIO for storage in the exercise.

• Nessie source connection: Explain why Dremio needs a catalog connection to query Iceberg tables and describe how the Nessie source is added and what appears afterward.
  Why it matters: The catalog connection makes the registered Iceberg tables visible in Dremio.
• MinIO storage connection settings: Describe the storage settings and connection properties used to let Dremio find the Iceberg files in this exercise’s S3-compliant MinIO storage.
  Why it matters: Finding a table in the catalog is not enough unless Dremio can also reach its stored files.
• Exercise authentication caveat: Distinguish this exercise’s simplified authentication and encryption settings from the production guidance stated in the section.
  Why it matters: The exercise’s simplified settings should not be mistaken for a production access-control recommendation.

Key terms: Add Source, Nessie, Iceberg catalog, Nessie endpoint URL, Storage settings, AWS root path, fs.s3a.path.style.access, fs.s3a.endpoint, dremio.s3.compat, MinIO

3.4.3 Querying Iceberg tables in Dremio

This section uses Dremio to query an Iceberg sales table, explains how file metadata can speed up queries, and shows how snapshots support time travel.

• Dremio sales table query: Describe what a query of the Iceberg sales table in Dremio returns and where that data came from.
  Why it matters: The query confirms that Dremio can read the ingested sales data as an Iceberg table.
• Metadata driven file pruning: Explain how Iceberg file metadata lets an engine such as Dremio skip files that cannot match a query.
  Why it matters: Skipping files that cannot contain a match reduces unnecessary data scans.
• Snapshot metadata and time travel: Describe what the snapshot metadata output represents and how snapshot IDs or timestamps can be used to query earlier table versions.
  Why it matters: Earlier table versions can help investigate errors and review historical data for auditing or compliance.

Key terms: Dremio SQL Editor, sales data, Iceberg table, MinIO, metadata-driven pruning, per-file statistics, sales_amount, metadata tables, snapshots, snapshot IDs

3.5 Creating a BI dashboard from your Iceberg tables

Introduces the steps for creating a BI dashboard from Iceberg data using Dremio and Apache Superset.

3.5.1 Starting Apache Superset

This section introduces Apache Superset and explains how to start, initialize, and log in before connecting it to Dremio.

• Superset startup and initialization: Describe the sequence for getting Superset ready: start it if needed, initialize it, then open its local web page and log in.
  Why it matters: These steps get Superset ready for the subsequent Dremio connection.
• Superset BI purpose: Explain what Superset is used for and what kind of sources it works with.
  Why it matters: Its purpose explains why Superset appears in the dashboard-building workflow.

Key terms: Apache Superset, initialize, metadata tables, roles, BI tool, interactive dashboards, SQL-based data sources

3.5.2 Connecting Superset to Dremio

This subsection shows how to connect Superset to Dremio and notes the protocol and encryption setting used by the local connection.

• Superset Dremio connection setup: Describe how to add Dremio as a database connection in Superset, test the connection, and save it.
  Why it matters: This connection lets Superset use Dremio as a data source.
• Connection protocol and encryption: Explain what protocol the Dremio URI uses and how the encryption setting differs between this local setup and production.
  Why it matters: The local connection setting should not be copied unchanged into production.

Key terms: Database Connections, Other, Dremio URI, Test Connection, Arrow Flight, UseEncryption=true

3.5.3 Creating a dataset from Iceberg tables

Shows how to create a Superset dataset from an Iceberg table so it can be used in charts and dashboards.

• Superset dataset creation: Describe how to create a Superset dataset from the Iceberg sales_data table through Dremio and state what the dataset enables.
  Why it matters: The dataset is the step that makes the Iceberg table available for visualization in Superset.

Key terms: Create Dataset, Dremio, nessie.sales.fashion_sales, Iceberg table

3.5.4 Building charts and dashboards

Shows how to turn an Iceberg-backed Superset dataset into charts and a dashboard, then explains what the exercise demonstrates about consuming Iceberg data.

• Sales chart configurations: Describe how the fashion_sales dataset is used to make a sales-over-time line chart and a sales-by-category pie chart in Superset.
  Why it matters: The different chart settings answer different questions using the same sales data.
• Sales dashboard assembly: Describe how the saved Sales by Day and Sales by Category charts are combined into a Sales Analytics dashboard.
  Why it matters: The dashboard makes the two views of Iceberg sales data available together.
• Familiar and modular consumption: Explain what using Superset with these Iceberg tables demonstrates about existing BI workflows and tool choice.
  Why it matters: The exercise illustrates that adopting Iceberg need not force a new way of consuming data or lock every step to one tool.
• Container volume cleanup: Explain why the environment cleanup uses Docker’s -v flag.
  Why it matters: Cleaning up container volumes recovers storage after the hands-on exercise.

Key terms: fashion_sales, Line Chart, Pie Chart, sales_date, category, SUM(sales_amount), Sales by Day, Sales by Category, Sales Analytics, Save & Publish

3.6 Summary