Object store
An object store is a storage system that stores and retrieves data as independently addressable objects containing content and associated metadata.
Architecting an Apache Iceberg Lakehouse
Object storage is a storage system that manages data as self-contained objects addressed through a service interface rather than a hierarchical filesystem.
- How it is used
- Anchors the lakehouse storage layer by providing scalable, durable, API-accessible storage for analytical data and metadata.
- How deep it goes
- Explains object structure, flat namespaces, metadata, distribution, and storage-compute tradeoffs, while stopping short of vendor-specific implementation detail.
- Examples
- Uses Amazon S3, MinIO, path-shaped keys, Parquet files, and a hybrid regulated-bank architecture as concrete examples.
- What you can do after
- Evaluate object-storage options against scalability, latency, resilience, cost, compliance, and S3-compatible lakehouse integration requirements.
Read how Architecting an Apache Iceberg Lakehouse explains object store →
Amazon Web Services in Action, Third Edition: An in-depth guide to AWS
An object store is a storage system that manages data as objects identified by keys rather than as files in a hierarchical filesystem.
- How it is used
- Stores backups, static website content, and application uploads without keeping those objects on application servers.
- How deep it goes
- Defines object components and buckets, then works through CLI website setup and SDK image access; distributed-store internals are not detailed here.
- Examples
- Uses a static HTML website, an image gallery, and access-log processing to show storing and retrieving objects.
- What you can do after
- Operate/configure Amazon S3 buckets for object uploads and static hosting, while checking public access and usage-based costs.
Machine Learning Platform Engineering
An object store is a storage system that manages data as objects identified by keys, typically including their content and metadata.
- How it is used
- Stores prepared ML datasets outside the workstation so pipelines can access them and MLflow can track their paths.
- How deep it goes
- Walks through bucket creation, client configuration, uploads, and dataset-path logging; leaves object-store internals unexplained.
- Examples
- Uses run-specific training, test, and reference CSVs plus uploaded object-detection images and labels.
- What you can do after
- Operate/configure MinIO buckets and boto3 uploads, then record stored dataset paths in MLflow runs.
Read how Machine Learning Platform Engineering explains object store →
Data Pipelines with Apache Airflow, Second Edition
Object Storage is an Airflow feature that provides a unified syntax for interacting with object-storage systems and transferring data between them.
- How it is used
- Stages data for transfers and processing between Airflow tasks, databases, containers, and AWS services.
- How deep it goes
- Works through S3 copying and hook-based transformation, but does not demonstrate a storage-independent object-store API.
- Examples
- Uses an Airbnb listings pipeline with MinIO and an MNIST pipeline that converts S3 data for SageMaker.
- What you can do after
- Apply it via S3Hook and S3CopyObjectOperator to prepare stored data, then verify individual tasks with `airflow tasks test`.
Read how Data Pipelines with Apache Airflow, Second Edition explains object store →
LLMs in Production
An object storage system stores data as independently addressable objects distributed across storage infrastructure.
- How it is used
- Uses object storage to hold large LLM assets reliably before service startup and model loading.
- How deep it goes
- Explains fragmentation, federation, replication, availability, and retrieval costs, then stops short of detailed storage implementation.
- Examples
- Uses GCS, S3, cloud buckets, compiled LLMs, and fused buckets as concrete storage and retrieval examples.
- What you can do after
- Evaluate storage choices for large model assets while accounting for reliability, retrieval speed, and boot-time loading.
Related concepts
- Objectuses · java.lang.Object is the root class of Java’s reference-type hierarchy and defines methods inherited by Java reference types.
- static siteused by · A static site is a website primarily devoted to presenting content through fixed or prebuilt pages rather than pages dynamically generated for each request.
- contractuses · An API contract is a specification of the operations, data, and behavioral expectations that govern interactions between software components.
- MinIOincludes · MinIO is an open-source object-storage server that provides an Amazon S3-compatible API for storing unstructured data.
- data lakeused by · A data lake is a repository that stores large volumes of data in raw or lightly processed form and varied formats for later processing and analysis.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.