Hybrid storage model

As explained in Architecting an Apache Iceberg Lakehouse

A hybrid storage architecture combines multiple storage environments, such as cloud object storage and on-premises storage, in one data platform.

A hybrid storage model combines cloud and on-premises environments in one data platform. It keeps selected datasets locally for latency, cost, sovereignty, or compliance while using cloud storage for analytics, elasticity, or long-term retention. section 4.3.1

Design requirements should make the boundary between environments explicit:

• Expose a consistent object-store interface across cloud and on-premises systems. section 4.3.1
• Provide unified metadata and catalog integration across heterogeneous storage backends. section 4.3.1
• Specify access controls, auditability, encryption, retention, network isolation, and capacity-management responsibilities. section 4.3.1
• Expect greater monitoring, troubleshooting, and operational complexity than a fully cloud or fully on-premises design. section 4.3.1

Banking lakehouse deployment

Hamerliva Bank uses Amazon S3 for elastic cloud analytics and MinIO clusters in European and US data centers for residency, air-gapped operation, and encrypted storage at rest. Both environments support its Iceberg lakehouse. section 4.5.1

Figure 4.6 The Hamerliva Bank architectural proposal shows how the components relate to one another.

Use the architectural proposal to trace how hybrid storage connects with ingestion, catalogs, compute, governance, and analytics components. section 4.5.1

How Architecting an Apache Iceberg Lakehouse covers it

How deep it goes

Defines boundary requirements and operational tradeoffs, then demonstrates a concrete S3–MinIO deployment without detailing storage internals.

Examples

Uses Amazon S3, on-premises MinIO clusters, regional data centers, air-gapped environments, and a banking lakehouse architecture.

What it assumes

Assumes storage and compliance requirements can be identified, while introducing hybrid boundary standards and their tradeoffs.

What you can do after

Evaluate storage requirements and design a hybrid architecture with explicit interfaces, governance controls, and operational safeguards.

How it connects

Connects cloud object storage, on-premises storage, data residency, metadata catalogs, and Iceberg lakehouses.

liveBookmanning.com

Read Architecting an Apache Iceberg Lakehouse on liveBook Get the book at manning.com

Related concepts

• MinIOuses · MinIO is an open-source object-storage server that provides an Amazon S3-compatible API for storing unstructured data.
• self-hostinguses · Self-hosting is deploying and operating software or services on computing infrastructure controlled or managed by the user or organization.
• cloud storageuses · Cloud storage is remotely managed storage delivered as a network-accessible service for persistently storing and retrieving data.
• role-based access controluses · Role-based access control is an authorization model that assigns permissions to roles and grants users or other principals access through their assigned roles.

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.