An object store manages data as self-contained objects rather than filesystem files. Each object combines content with metadata and a unique key, and clients retrieve it through service APIs, commonly the S3 API. This flat, key-based model separates storage from compute, enabling independent scaling and reducing the operational coupling of colocated storage. section 5.2.2 section 5.2
Object stores provide several architectural properties relevant to lakehouse platforms:
- Scale horizontally across storage nodes, using distribution, load balancing, replication, and fault tolerance to support large collections of objects. section 5.2.2
- Use keys that can resemble paths, such as logs/2024/01/file.parquet, without requiring nested filesystem directories. section 5.2.2
- Attach timestamps, sizes, permissions, and user-defined tags to objects, supporting search, lifecycle rules, and access control. section 5.2.2
- Trade lower operational overhead and independent scaling for network-dependent access with higher and more variable latency than block storage. section 5.4 section 5.2
In an Apache Iceberg lakehouse, an object store holds Parquet data files and Iceberg metadata. Engines use the S3 API to access these assets, while Iceberg’s transactional metadata helps coordinate concurrent reads and writes. Immutability, consistency, and API compatibility therefore matter when evaluating storage backends. section 5.3 section 5.4.1
Hybrid lakehouse storage
A regulated bank can place elastic analytics workloads on Amazon S3 while keeping residency-sensitive data on encrypted, on-premises MinIO clusters. This combines independent scaling with local control and S3-compatible access. section 4.5.1
Figure 5.6 Files are stored as blobs (binary large objects) alongside an object that holds the metadata and ID used to access them. A user requests a file by key, and the system uses the metadata to locate and retrieve it.
Use the object diagram to trace how a request for a key leads from object metadata and identifiers to the stored data blob. section 5.2.2
