MinIO

As explained in Data Pipelines with Apache Airflow, Second Edition

MinIO is an object-storage server that provides an Amazon S3-compatible API.

MinIO is an object-storage server that supports the Amazon S3 API. In Airflow pipelines, it provides a containerized storage endpoint for moving data between systems. section 8.3.1 section 13.1

A typical flow queries PostgreSQL, writes the result to MinIO, and then reads the stored data for processing with pandas. MinIO can also sit among REST APIs, file shares, databases, Airflow, and downstream applications. section 8.3.1 section 13.1

MinIO is useful when a reproducible pipeline needs:

• An S3-compatible object store running alongside local services. section 8.3.1
• A handoff point between extraction and transformation tasks. section 8.3.1
• A storage component that can connect data ingestion with later database and web outputs. section 13.1

Airbnb listings pipeline

A Docker Compose setup runs PostgreSQL with Airbnb listings, MinIO, and Airflow. The workflow queries the database, exports records to S3-compatible storage, and processes them with pandas. section 8.3.1

Figure 13.1 Our Docker Compose file creates several services. Our task is to load data from the REST API and file share and then transform it so we can eventually view the fastest method of transportation on the resulting web page.

Shows MinIO among the Dockerized services that support extraction, transformation, database storage, and web presentation. section 13.1

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes

Defines MinIO and demonstrates its containerized pipeline role, without detailing storage internals.

Examples

Uses Airbnb listings and New York transportation pipelines to demonstrate MinIO alongside databases, APIs, file shares, and Airflow.

What you can do after

Operate/configure MinIO through Docker Compose as an S3-compatible storage endpoint in a multi-service data pipeline.

How it connects

Connects MinIO with Amazon S3, Airflow, Docker Compose, PostgreSQL, pandas, REST APIs, file shares, and web applications.

liveBookmanning.com

Read Data Pipelines with Apache Airflow, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Architecting an Apache Iceberg LakehouseMinIO is an S3-compatible object-storage system that can be deployed on premises or in other infrastructure.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringMinIO is an open-source object-storage server that provides an Amazon S3-compatible interface for storing unstructured data.

Compare how 3 books explain MinIO →

Related concepts

• object storekind of · An object store is a storage system that stores and retrieves data as independently addressable objects containing content and associated metadata.
• S3 APIimplements · The S3 API is an object-storage interface for storing, retrieving, and managing objects and buckets according to Amazon S3 conventions.
• cloud storageuses · Cloud storage is remotely managed storage delivered as a network-accessible service for persistently storing and retrieving data.
• Docker Composeconfigured by · Docker Compose is a tool for defining and running applications composed of multiple containers through declarative configuration.

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.