Chapter 5: Selecting the storage layer

Introduction

The introduction explains why storage choices matter to an Iceberg lakehouse and identifies Parquet and the S3 API as foundational standards.

• Storage Layer’s Foundational Role: Explain why the storage layer is a consequential architectural choice for an Apache Iceberg lakehouse.
  Why it matters: Storage decisions shape how well the rest of the lakehouse can operate.
• Parquet’s Analytical Fit: Explain why the introduction identifies Parquet as suited to Iceberg workloads.
  Why it matters: The file format is part of how storage supports analytical use.
• S3 API Compatibility Role: Describe why the introduction treats the S3 API as a foundational object-storage standard.
  Why it matters: A broadly compatible storage interface gives architects more deployment options.

Key terms: storage layer, reliability, scalability, cost efficiency, Parquet, columnar structure, analytics-oriented workloads, S3 API, object storage, compatibility

5.1 Storage requirements

This section explains how to turn an organizational audit into storage requirements before comparing options for an Iceberg lakehouse.

• Requirements-first storage selection: Explain why storage selection should begin with organizational goals rather than a preferred technology or vendor.
  Why it matters: Clear requirements give you a basis for judging storage options beyond vendor claims.
• Audit-derived storage dimensions: Describe how stakeholder audit findings become concrete storage requirements and name the dimensions this section identifies.
  Why it matters: Audit findings make the storage comparison specific to the people and constraints the lakehouse must serve.

Key terms: storage layer, requirements, Apache Iceberg lakehouse, audit process, performance, security, integrity, operational and cost considerations

5.1.1 Performance requirements for file retrieval

This subsection explains which factors shape file-access performance in an Iceberg lakehouse and how to evaluate storage against real workloads.

• Engine and storage performance roles: Explain why raw storage latency is usually not the main determinant of interactive lakehouse performance, while storage performance still matters for large scans and ingestion.
  Why it matters: It prevents you from choosing storage solely by its advertised latency when the engine does much of the work of delivering fast queries.
• Concurrency and workload access patterns: Describe how concurrent demand and differing read and write patterns affect storage suitability.
  Why it matters: A storage system that looks fast for one task may slow down when real workloads overlap.
• File organization and read reduction: Explain how Iceberg’s file-selection optimizations and columnar compression reduce data read, and why predictable storage service still matters.
  Why it matters: Storage performance depends partly on how many bytes the engine needs to fetch, not just how quickly storage can serve each request.
• Workload-based storage performance testing: Describe how to evaluate storage performance using representative workloads rather than advertised throughput alone.
  Why it matters: Testing realistic behavior exposes slow requests and contention that a headline throughput figure may hide.

Key terms: raw storage latency, in-memory execution, local caching, adaptive buffering, object storage, concurrency, access patterns, sequential reads, concurrent random writes, read contention

5.1.2 Security requirements

This section explains the security requirements to assess when choosing storage for an Iceberg lakehouse.

• Storage as a trust boundary: Explain why storage security matters even when the catalog or query engine provides access controls.
  Why it matters: Storage security must be evaluated as a foundation, not assumed to follow from catalog permissions.
• Identity-based storage permissions: Describe the access controls a storage solution should support and how their enforcement differs between object and block storage.
  Why it matters: Permission models determine who can directly read or change stored data.
• Encryption and key control: Explain the encryption and key-management capabilities to consider, particularly for regulated organizations.
  Why it matters: Encryption and key control can be necessary to meet regulated-data obligations.
• Storage access audit logs: Describe what storage activity should be logged and why those logs must be retained.
  Why it matters: Useful, retained logs help investigate suspicious activity and security incidents.
• Storage location and jurisdiction: Explain why storage location belongs in a security evaluation and describe the cloud versus on-premises tradeoff raised here.
  Why it matters: Where data is stored affects both compliance and recovery planning.

Key terms: storage layer, foundational trust boundary, fine-grained access controls, fine-grained access control, bucket-level, prefix-level, identity and access management policies, filesystem permissions, host-level isolation, server-side encryption

5.1.3 Integrity requirements

This section explains the storage-layer capabilities needed to keep Iceberg lakehouse data accurate, recoverable, and accessible despite failures.

• Storage durability guarantees: Explain how durability protects stored data from corruption or loss and why a storage system’s redundancy and recovery approach matters when evaluating it.
  Why it matters: A storage choice needs to match how much data loss or disruption the platform can tolerate.
• Coordinated Iceberg table recovery: Explain why recovering an Iceberg table requires restoring matching data files and table metadata, rather than relying on storage-level snapshots alone.
  Why it matters: A backup is useful only if it can restore a working Iceberg table, not just a collection of files.
• Corruption detection through validation: Describe how checksumming and validation during reads and writes support storage integrity.
  Why it matters: Early detection makes corrupted data less likely to contaminate later work.
• Availability through storage redundancy: Explain how high availability complements integrity by maintaining access when storage components fail.
  Why it matters: Even intact data cannot support ingestion or analytics if users and services cannot reach it.

Key terms: data integrity, durability, replication, consistency, recovery, storage-level snapshots, data files, table metadata, catalog, ACID guarantees

5.1.4 Cost and operational overhead requirements

This section explains how to assess a storage layer’s total cost and operational burden alongside its technical requirements.

• Total storage ownership costs: Explain why storage costs include more than capacity and how cloud and on-premises systems incur those costs differently.
  Why it matters: Comparing capacity prices alone can hide substantial long-term costs.
• Operational effort and scaling: Describe how management effort, automation, and the labor needed to scale affect a storage choice.
  Why it matters: A storage system that is cheap to buy may still demand substantial team time as it grows.
• Interoperability and integration costs: Explain how standard versus proprietary storage interfaces affect operational cost and flexibility.
  Why it matters: Integration effort can turn an apparently suitable storage system into a costly long-term constraint.

Key terms: total cost of ownership, cloud-based object storage, on-premises storage, operational overhead, automation, observability tools, self-service capabilities, interoperability, standard APIs and protocols, proprietary interfaces

5.2 Block vs. object storage

Introduces block and object storage as two Iceberg-compatible architectures and explains their main tradeoffs for a lakehouse.

• Block storage tradeoffs: Explain how block storage organizes data and why its performance benefits come with scaling and operational tradeoffs in a lakehouse.
  Why it matters: It helps you weigh fast access against the cost and effort of operating the storage layer.
• Object storage advantages: Describe how object storage represents data and why its separation of storage from compute makes it a common choice for modern lakehouses.
  Why it matters: It explains why an object store may better suit a lakehouse that needs flexible growth and simpler operations.

Key terms: block storage, fixed-size chunks, colocated with compute, object storage, objects, metadata, unique identifier, separates compute from storage

5.2.1 Block storage

This section explains how block storage works, why it suited Hadoop and some performance-sensitive workloads, and what tradeoffs it presents for an Iceberg lakehouse.

• Blocks and file assembly: Explain how block storage divides data into evenly sized blocks and how those blocks become files and directories for applications.
  Why it matters: This is the foundation for understanding the storage architecture used by HDFS.
• Replication and fault tolerance: Describe how replicas help distributed storage stay available when components fail, and explain the cost tradeoff of keeping more copies.
  Why it matters: Replication choices affect reliability as well as storage cost.
• Locality and consistent IOPS: Explain why mounting block storage close to compute can benefit certain workloads, including the role of consistent IOPS.
  Why it matters: It explains why block storage can remain attractive despite its management costs.
• Legacy fit and operational costs: Explain why existing Hadoop or on-premises installations may keep block storage when adopting Iceberg, and contrast that practicality with its operational and scaling costs.
  Why it matters: A practical storage choice must account for existing infrastructure as well as ongoing operational effort.

Key terms: block storage, blocks, filesystem, replication factor, fault tolerance, replicas, compute nodes, low latency, input/output operations per second (IOPS), HDFS

5.2.2 Object storage

Explains why object storage is the default for many Iceberg lakehouses, how it organizes and manages objects, and which costs and access tradeoffs architects must consider.

• Object organization and access: Explain how object storage packages data and how users access an object without relying on a directory hierarchy.
  Why it matters: This model explains both how Iceberg reaches stored files and why familiar-looking paths need not be filesystem directories.
• Distributed scale and durability: Describe how distributed object stores support large lakehouse deployments and which built-in management capabilities the section highlights.
  Why it matters: These capabilities make object storage practical for the volume and operational demands of a lakehouse.
• Metadata and access controls: Explain how object metadata and built-in security features support managing and protecting lakehouse data.
  Why it matters: Storage selection must account for governance and security as well as capacity.
• Pricing and deployment options: Describe the pricing and operational appeal of cloud object storage and the control offered by on-premises options.
  Why it matters: The choice affects both operating effort and where an organization can keep its data.
• Latency and consistency tradeoffs: Explain why architects must consider object-store latency and consistency even though Iceberg supports object storage.
  Why it matters: Ignoring per-request delays or visibility behavior can undermine low-latency queries and reliable pipelines.

Key terms: object, metadata, unique identifier, key, flat namespace, S3 protocol, autosharding, load balancing, horizontal scaling, replication

5.3 Storage layer standards

Introduces Parquet and the S3 API as widely adopted storage-layer standards that help Iceberg work across tools, platforms, engines, and clouds.

• Parquet and S3 API roles: Explain that Parquet is a file format commonly used by Iceberg tables, while the S3 API is a widely supported object-storage interface, and why their adoption matters for compatibility.
  Why it matters: These shared standards help you evaluate whether storage systems will fit an Iceberg lakehouse.

Key terms: Parquet file format, S3 API, storage layer, interoperability

5.3.1 Apache Parquet

This section explains why Parquet is the usual physical file format for Iceberg tables and how its columnar storage works with Iceberg metadata to make analytical reads efficient.

• Columnar reads and row-group skipping: Explain how Parquet’s columnar row groups reduce analytical scan work: an engine can read only needed columns, benefit from per-column compression, and use row-group statistics to skip groups that cannot match a filter.
  Why it matters: These are the file-level reasons Parquet can scan analytical data efficiently.
• Parquet statistics in Iceberg metadata: Trace how Iceberg uses statistics from Parquet files in its own metadata so query planning can rule out irrelevant files without first inspecting each file.
  Why it matters: Iceberg can avoid opening irrelevant files before Parquet’s within-file optimizations come into play.
• Iceberg-controlled schema evolution: Distinguish the schema stored in a Parquet file from Iceberg’s table-level schema management, and explain why field IDs let files written under different schemas coexist.
  Why it matters: It prevents attributing Iceberg’s safe table-level schema changes to Parquet alone.
• File sizing over row-group tuning: Explain why the section favors file-level parallelism, metadata pruning, and engine defaults over manually tuning Parquet row groups or imposing a universally large file size.
  Why it matters: It focuses performance work on factors the section says usually matter more than fine-grained Parquet layout settings.
• Parquet as the practical standard: Justify why Parquet is the usual Iceberg data-file format despite Iceberg allowing multiple formats.
  Why it matters: It explains why an architecture can permit several file formats yet commonly choose Parquet.

Key terms: columnar, row groups, column chunks, predicate pushdown, file-level statistics, manifest metadata, file pruning, query planning, file schema, table schema

5.3.2 The S3 API

This section explains why the S3 API is a common interface for Iceberg object storage, how its object model fits Iceberg files, and why compatibility claims need validation.

• S3 API interoperability: Explain how the S3 API lets Iceberg and other tools access object storage across vendors through a common interface.
  Why it matters: A common storage interface gives platform teams more choice of object store.
• Object model and Iceberg files: Describe the S3 API’s bucket-and-key object model and explain why independently addressable objects fit Iceberg’s immutable data and metadata files.
  Why it matters: The way S3 addresses objects helps explain why multiple engines can access an Iceberg table’s files.
• Shared S3 tooling ecosystem: Explain how S3-compatible operational tools and API features can help teams manage storage deployments.
  Why it matters: Reusing S3-compatible tools can reduce integration work across storage deployments.
• S3 compatibility validation: Justify checking an object store’s S3 implementation under production workloads rather than accepting a compatibility claim alone.
  Why it matters: A compatibility label alone does not establish that a store will work well for an Iceberg deployment.

Key terms: S3 API, S3 API–compatible object store, interoperability, flat object namespaces, bucket, unique keys, immutable files, data files, metadata files, S3 endpoints

5.4 Storage solutions

Introduces storage-solution selection by explaining how colocated and separate storage architectures affect reliability, latency, and scaling for an Iceberg lakehouse.

• Storage architecture tradeoffs: Compare colocated and separate storage in terms of replication and recovery, latency, operational overhead, and the ability to scale compute and storage independently.
  Why it matters: These tradeoffs help you match a storage architecture to your workload rather than choose by vendor name alone.
• Write visibility and consistency: Explain why the visibility of newly written files matters when evaluating distributed storage for an Iceberg lakehouse.
  Why it matters: Coordinated reads and writes depend on storage making new files visible reliably.

Key terms: colocated storage, separate storage, software-managed replication, independent scaling, consistency, write visibility, transactional guarantees

5.4.1 Vendor comparison summary

This subsection frames storage-vendor comparison as a requirements-led starting point for selecting Iceberg lakehouse storage, including possible hybrid choices.

• Requirements-led storage comparison: Explain why an Iceberg storage choice should be evaluated against the organization’s performance, governance, and operational requirements rather than by a vendor’s features alone.
  Why it matters: A requirements-led comparison helps you choose storage that fits your lakehouse rather than a generic best option.
• Comparison limits and hybrid storage: Describe how the vendor summary can be used in an initial evaluation, and explain when multiple storage backends may be appropriate.
  Why it matters: The summary helps form a shortlist without prematurely ruling out a combination of backends.

Key terms: performance, governance, operational requirements, scalability, ecosystem integration, ease of management, hands-on evaluation, multiple storage backends, immutability, consistency

5.4.2 Hadoop

This section explains how HDFS works, why it can support an incremental move to Iceberg, and where its limitations make it a weaker choice for a new lakehouse.

• HDFS storage model: Describe how HDFS stores large files and why it has suited on-premises big data workloads.
  Why it matters: HDFS’s storage model helps explain both its continued value in existing clusters and its operational demands.
• Incremental Iceberg adoption: Explain why an organization with an established Hadoop cluster might keep HDFS while adopting Iceberg.
  Why it matters: An existing HDFS investment need not prevent a gradual move to Iceberg tables.
• HDFS suitability limits: Distinguish situations where HDFS remains useful from those where its limitations make it less suitable.
  Why it matters: Storage selection depends on whether the value of an existing HDFS deployment outweighs its limits for the intended lakehouse.

Key terms: HDFS, fixed-size blocks, replication, distributed filesystem, Iceberg, Parquet, format agnostic, metadata and table management, elastic scaling, operational overhead

5.4.3 Amazon S3

This section explains why Amazon S3 is a common storage choice for AWS-based Iceberg lakehouses, what performance, cost, and security considerations it brings, and how S3 table buckets differ from general-purpose buckets.

• S3 as Iceberg storage: Explain how S3 stores Iceberg tables and why it fits AWS-based lakehouses. Describe how compute engines access those tables and how Iceberg metadata works with S3 objects to support concurrent use.
  Why it matters: It explains why S3 can serve as the shared storage foundation for an AWS-based Iceberg lakehouse.
• S3 consistency and commits: Explain why S3 performance still calls for care in commit workflows and cleanup jobs, and how Iceberg mitigates the stated consistency concerns.
  Why it matters: Strong general performance does not remove the need to check behavior under demanding write workloads.
• S3 cost drivers: Describe the usage patterns and pricing factors that can raise S3 costs, and identify the kinds of controls the section suggests.
  Why it matters: A storage choice that performs well can still become expensive under the wrong access patterns.
• S3 security controls: Describe the S3 capabilities the section identifies for access control, data protection, auditing, and policy enforcement.
  Why it matters: These controls help an architect assess whether S3 meets enterprise data-protection requirements.
• S3 table bucket capabilities: Distinguish S3 table buckets from general-purpose buckets and describe the Iceberg-specific management, access, and security features stated here.
  Why it matters: Table buckets offer a more managed, Iceberg-specific S3 option without limiting access to AWS query engines.

Key terms: Amazon S3, S3 buckets, Iceberg metadata, data files, object immutability, eventual consistency, listing, overwriting, atomic pointer updates, file-level tracking

5.4.4 Google Cloud Storage

This section describes how Google Cloud Storage supports Iceberg lakehouses through consistent object storage, storage classes, analytics performance, and security controls.

• Consistency for Iceberg operations: Explain how GCS organizes and exposes data, and why its consistency guarantee matters for Iceberg metadata commits and snapshot reads.
  Why it matters: File visibility affects how confidently an Iceberg lakehouse can handle commits, reads, ingestion, and cleanup.
• Storage classes by access pattern: Describe how GCS storage classes let an Iceberg lakehouse balance cost and performance according to access frequency and retention.
  Why it matters: Matching storage class to usage can control the cost of retaining Iceberg data.
• Analytics performance and integration: Explain why GCS is presented as a good storage fit for Iceberg analytics workloads, particularly within Google’s analytics ecosystem.
  Why it matters: Storage choice should account for both analytics throughput and the services an organization already uses.
• Encryption access and auditing: Describe the GCS controls the section identifies for protecting and governing stored data.
  Why it matters: These controls help assess whether GCS meets a lakehouse’s security and compliance requirements.

Key terms: buckets, objects, RESTful API, strong read-after-write consistency, metadata commits, snapshot reads, Standard, Nearline, Coldline, Archive

5.4.5 Azure Blob Storage and ADLS

Compares Azure Blob Storage and ADLS as Iceberg storage backends, including their access, performance, security, and cost tradeoffs, and introduces OneLake.

• Blob and ADLS tradeoffs: Distinguish Blob Storage’s flat object namespace from ADLS Gen2’s hierarchical namespace and filesystem semantics. Explain how workload patterns and governance needs guide the choice of backend for Iceberg.
  Why it matters: The better Azure storage choice depends on how the lakehouse accesses and governs its files.
• Layered ADLS access control: Explain the distinct roles of Azure AD and POSIX-style ACLs in controlling access to ADLS data.
  Why it matters: Separating identity verification from file permissions clarifies why ADLS suits granular governance requirements.
• Blob tiers and workflows: Describe how Blob Storage’s access tiers and Azure service integrations can support Iceberg storage operations.
  Why it matters: These capabilities affect the cost and operational fit of Blob Storage beyond its basic ability to hold Iceberg files.
• OneLake interoperability options: Describe OneLake’s relationship to Microsoft Fabric and the interoperability options noted in this section.
  Why it matters: OneLake offers a way to connect Fabric to Iceberg and existing external storage without requiring a physical data move.

Key terms: Azure Blob Storage, Azure Data Lake Storage Gen2, flat namespace, hierarchical namespace, filesystem semantics, Azure AD, POSIX-style ACLs, authentication, authorization, Hot

5.4.6 MinIO

This section evaluates MinIO as an S3-compatible storage option for Iceberg lakehouses and explains the performance, protection, security, and operational features that shape its fit.

• MinIO deployment fit: Explain why MinIO can suit Iceberg deployments outside the public cloud and what tradeoff that choice entails.
  Why it matters: Deployment constraints can make MinIO a better fit than a public cloud storage service.
• S3 interoperability and performance: Describe how MinIO’s S3 API compatibility lets Iceberg tools use it and what performance benefits the section attributes to it.
  Why it matters: Compatibility and retrieval speed affect whether existing Iceberg workflows can move to MinIO without major changes.
• Erasure-coded data protection: Explain how erasure encoding protects stored data and why it can use less space than full replication.
  Why it matters: This explains how MinIO can provide durability without the space cost of multiple full copies.
• Inline compression effects: Describe when inline compression happens and how it affects storage and reads.
  Why it matters: Compression can lower storage use and read I/O without adding a separate processing job.
• Object locking for retention: Explain how object locking supports immutable retention and distinguish its governance and compliance modes.
  Why it matters: The lock mode determines how strongly retained records are protected against changes.
• MinIO security and management: Describe the security controls and management capabilities the section attributes to MinIO.
  Why it matters: Security and manageable operations matter when running a lakehouse outside a managed public cloud service.

Key terms: MinIO, on-premises, hybrid environments, edge environments, S3 API, metadata commits, data file retrieval, erasure encoding, erasure coding, parity fragments

5.4.7 Ceph

This subsection explains how Ceph can provide S3-compatible storage for Iceberg lakehouses and what teams must weigh when operating it.

• Ceph storage architecture: Explain how Ceph combines object, block, and file storage, and identify the component that makes it usable as an S3-compatible backend for Iceberg tables.
  Why it matters: The RGW interface is the link between Ceph’s storage system and Iceberg’s need for object storage.
• Distributed durability and recovery: Describe how Ceph protects data across nodes and responds to hardware failures.
  Why it matters: These capabilities make Ceph relevant to teams that need uptime and data integrity across a multinode cluster.
• Performance and operating tradeoffs: Explain why Ceph’s performance and suitability depend on deployment choices and the team operating it.
  Why it matters: Choosing Ceph means committing to cluster design and ongoing operational expertise, not just choosing an S3-compatible endpoint.
• S3 compatibility limits: Explain why Ceph’s S3 compatibility helps Iceberg integration but still calls for production validation.
  Why it matters: An S3-compatible API is useful, but testing the intended Iceberg workload remains important before production use.
• Configurable security responsibilities: Describe Ceph’s stated security capabilities and the work needed to maintain a strong security posture.
  Why it matters: Security features alone do not remove the organization’s responsibility to configure and enforce them.

Key terms: Ceph, RADOS, RBD, CephFS, RGW, S3-compatible API, distributed replication, erasure coding, self-healing, rebalancing

5.4.8 NetApp StorageGRID

This section evaluates NetApp StorageGRID as an S3-compatible storage option for Iceberg lakehouses, emphasizing lifecycle policies, workload fit, and compliance.

• StorageGRID deployment fit: Explain why StorageGRID can serve as an Iceberg lakehouse’s object storage layer, particularly when an organization needs on-premises or hybrid deployment.
  Why it matters: StorageGRID’s fit depends on both Iceberg compatibility and where an organization must run its storage.
• Policy driven data lifecycle: Describe what StorageGRID lifecycle policies control and why they suit Iceberg datasets with multiple snapshots.
  Why it matters: Policy-based lifecycle management helps align storage cost and retention with the needs of Iceberg datasets.
• Analytical workload performance: Distinguish StorageGRID’s strengths in large-object analytical workloads from its latency limitation, and describe the durability features named in the section.
  Why it matters: Its performance strengths should be matched to analytical rather than latency-sensitive transactional workloads.
• WORM compliance and immutability: Explain how WORM protects retained records and why StorageGRID’s immutability and security features matter to regulated organizations.
  Why it matters: Tamper-resistant retention and auditability can be deciding storage requirements in regulated settings.

Key terms: StorageGRID, S3-compatible interface, hybrid cloud, intelligent data lifecycle policies, object placement, retention, tiering, snapshot-based model, large object workloads, high-throughput ingestion

5.4.9 Everpure

This section explains when Everpure FlashBlade is a suitable storage layer for an Iceberg lakehouse, emphasizing performance, operational simplicity, and enterprise security.

• FlashBlade performance and fit: Explain how FlashBlade serves Iceberg table storage and why its performance can justify a premium solution for demanding workloads.
  Why it matters: This connects a storage product’s capabilities to the workloads and priorities that could make it worth choosing.
• Turnkey storage operations: Describe how Everpure’s appliances and Pure1 reduce the work of operating a lakehouse storage layer.
  Why it matters: Storage selection affects how much time platform teams spend maintaining infrastructure.
• Enterprise storage security controls: Describe the FlashBlade capabilities the section associates with regulated environments.
  Why it matters: A storage layer for regulated data must support access control and governance as well as performance.
• LDAP and external identity providers: Distinguish LDAP-based directory services from external identity providers and explain why organizations may connect them.
  Why it matters: The distinction clarifies how storage access can fit into an organization’s existing and cloud-based identity systems.

Key terms: FlashBlade, S3 API, high-concurrency environments, small-object operations, turnkey appliances, Pure1, administrative overhead, encryption at rest, secure multitenancy, access controls

5.4.10 Dell ECS

This subsection explains when Dell ECS is a suitable object storage backend for an Iceberg lakehouse and the resilience, performance, and governance considerations that shape its use.

• ECS fit for Iceberg: Explain how ECS can serve an Iceberg lakehouse through its S3-compatible object storage, particularly when storage must remain on-premises or span a hybrid environment.
  Why it matters: S3 compatibility lets organizations consider ECS without giving up an object-storage backend for Iceberg.
• Resilience and performance planning: Describe the capabilities that support ECS at large scale and explain why an Iceberg deployment still needs performance planning.
  Why it matters: Storage capabilities alone do not guarantee responsive Iceberg reads as a deployment grows.
• Tenant security and compliance: Explain how ECS helps organizations separate tenants and enforce security and retention requirements.
  Why it matters: These controls can make an on-premises or hybrid storage choice viable where isolation and regulatory compliance are mandatory.

Key terms: Dell ECS, S3 API, object storage, hybrid, on-premises, data sovereignty, georeplication, erasure coding, horizontal scaling, storage pools

5.4.11 Wasabi

This section evaluates Wasabi as a cost-predictable, S3-compatible storage backend for Iceberg lakehouses and identifies where its capabilities and limits fit.

• Predictable storage pricing: Explain how Wasabi’s pricing model can reduce cost uncertainty for Iceberg workloads with frequent storage access.
  Why it matters: Storage access patterns can affect the total cost of a lakehouse, not just the amount of data stored.
• S3 compatibility and modular fit: Describe how Wasabi fits into an Iceberg lakehouse through S3-compatible access and why other services must be supplied separately.
  Why it matters: S3 compatibility makes Wasabi usable with familiar engines, while its storage-only focus shapes the rest of the architecture.
• Performance and security tradeoffs: Distinguish Wasabi’s stated reliability and core security features from its latency and access-control limitations.
  Why it matters: A predictable price does not remove the need to check workload performance and security requirements.

Key terms: flat, per-terabyte monthly fee, data egress, API requests, table scans, metadata reads, snapshot rewrites, S3 API, storage backend, modular architecture, compute and catalog services

5.5 Selecting storage based on requirements

This section frames storage selection as a tradeoff across organizational needs and constraints rather than a separate choice for each Iceberg use case.

• Storage selection across use cases: Explain why an organization will usually choose a storage platform that serves its broad range of Iceberg use cases, weighing requirements against its existing environment and constraints rather than selecting a different system for each use case.
  Why it matters: It keeps storage decisions grounded in what the organization can operate and use across its lakehouse.

Key terms: storage platform, operational and financial overhead, cloud strategy, compliance boundaries, existing tooling

5.5.1 Performance requirements

This subsection shows how different Iceberg workloads call for different storage performance profiles and illustrates how to match storage options to those needs.

• Workload-based performance evaluation: Explain why storage performance must be evaluated against workload access patterns and user expectations rather than a single average metric.
  Why it matters: The right storage choice depends on the workload’s actual bottlenecks, not a general claim that a system is fast.
• Latency-sensitive storage choices: Describe why interactive dashboards and scheduled GPU training jobs need reliably low-latency access, and relate that need to the storage options discussed.
  Why it matters: Slow retrieval can be visible to dashboard users or leave expensive training resources idle.
• Batch and archival tradeoffs: Distinguish the performance priorities of overnight batch ETL from those of non-urgent regulatory queries, and give storage examples for each.
  Why it matters: Separating throughput-bound processing from non-urgent reporting prevents paying for low latency where it is not the priority.

Key terms: throughput, latency, concurrency, consistency, performance profile, interactive dashboards, high concurrency, low-latency object retrieval, machine learning feature pipelines, caching strategies

5.5.2 Security requirements

Shows how different security and compliance needs shape storage choices for an Iceberg lakehouse.

• Contextual security fit: Explain why storage security must be evaluated against an organization’s policies, audit needs, identity infrastructure, legal obligations, and risk tolerance—not just a feature list.
  Why it matters: A secure-looking feature list does not establish that a storage system meets your organization’s obligations.
• Sensitive and segmented access: Describe how the healthcare and internal R&D scenarios call for different access controls, and identify the storage options the section associates with each.
  Why it matters: The kind of access restriction needed helps narrow the storage options and exposes configuration trade-offs.
• Cloud audit and key controls: Explain why the public-cloud financial scenario points to Amazon S3 or Google Cloud Storage.
  Why it matters: The scenario requires evidence of data access and changes as well as control over encryption keys.
• Jurisdictional data placement: Explain how the cross-jurisdiction scenario affects storage selection and why Dell ECS or StorageGRID fit it.
  Why it matters: Security selection can depend on legal limits on data location and movement, not only on who can access it.

Key terms: access control, encryption, auditability, compliance standards, WORM, granular access policies, fine-grained ACLs, multitenant deployments, role-based access controls, customer-managed keys

5.5.3 Integrity requirements

This section shows how durability, recoverability, consistency, and retention needs guide storage selection for an Iceberg lakehouse.

• Iceberg and storage responsibilities: Explain why Iceberg’s metadata safeguards do not replace the durability and resilience required of the underlying storage system.
  Why it matters: A trustworthy Iceberg table depends on both its metadata safeguards and the storage holding its files.
• Geographic recovery and replicas: Describe why a multisite enterprise might select storage with georeplication and policy-driven data protection.
  Why it matters: A site failure need not make a geographically distributed lakehouse’s data unavailable.
• Immutable long-term retention: Explain how object locking and WORM capabilities address long-term retention and tamper-resistance requirements.
  Why it matters: Retention rules are more dependable when storage itself prevents premature alteration or deletion.
• Consistency during metadata churn: Explain why strong storage consistency matters when Iceberg tables undergo frequent metadata commits and schema changes.
  Why it matters: Frequently updated tables need storage behavior that does not expose stale or incomplete metadata.
• Durability at disconnected edges: Describe how erasure coding and self-healing address integrity needs in an edge deployment with intermittent connectivity.
  Why it matters: Intermittent connectivity should not cause edge-stored Iceberg data to be lost before it can synchronize.

Key terms: metadata consistency, snapshot model, storage backend, durability, recoverability, multisite redundancy, georeplication, policy-driven data protection, object replicas, object locking

5.5.4 Cost and operational requirements

This section shows how to choose Iceberg storage by weighing long-term cost and operational capacity against the needs of different organizations.

• Long-term storage value: Explain why storage decisions should maximize value over time rather than minimize initial expense, accounting for engineering effort, team capabilities, budget, and lifecycle expectations.
  Why it matters: The operational work your team can sustain is part of the real cost of storage.
• Operational capacity and control: Describe how the startup and enterprise scenarios lead to different storage choices based on available infrastructure staff and the need for centralized controls.
  Why it matters: The right amount of operational control depends on who will run the platform and at what scale.
• Funding and deployment constraints: Describe how unpredictable grant-funded usage and government deployment constraints affect storage selection in the two scenarios.
  Why it matters: Funding patterns and deployment restrictions can matter as much as storage pricing.

Key terms: cost and operational overhead, engineering time, lifecycle expectations, Wasabi, MinIO, Dell ECS, NetApp StorageGRID, Everpure, multitenancy, pay-as-you-go pricing

5.6 Summary