liveBook · Manning

Httpfs extension

As explained in DuckDB in Action

The httpfs extension is a DuckDB extension that enables querying files over HTTP(S) and accessing supported cloud storage providers.

The httpfs extension lets DuckDB query files hosted over HTTP(S) without downloading them locally. It also supports S3 and several other cloud storage providers. section 2.4

Use it to:

  • Install and load remote-file support with INSTALL httpfs; and LOAD httpfs;. section 2.4
  • Query CSV or Parquet files directly from remote URLs and use the CLI within data pipelines. section 2.6 section 10.4.1
  • Configure a TYPE S3 secret when accessing a private bucket. section 10.4.1

Enable remote access

Run INSTALL httpfs; followed by LOAD httpfs; in the DuckDB CLI, then use a remote dataset such as https://oedi-data-lake.s3.amazonaws.com/pvdaq/csv/systems.csv. DuckDB may autoload an installed extension when it can determine that the extension is needed. section 3.2.1 section 2.4

INSTALL httpfs;
LOAD httpfs;

Inspect the minimal installation and loading commands for httpfs. section 2.4

How DuckDB in Action covers it

How deep it goes
Explains installation, loading, autoloading, remote queries, and private S3 credentials without detailing transfer internals.
Examples
Uses INSTALL and LOAD commands, PVDAQ HTTP and S3 URLs, remote CSV files, and direct Parquet access.
What it assumes
Assumes basic DuckDB CLI use and requires installation plus loading, while public S3 objects need no extra credentials.
What you can do after
Operate DuckDB against remote CSV and Parquet files, and configure credentials for private S3 access.
How it connects
Connects DuckDB extensions with HTTP(S), S3, Parquet, CSV, cloud storage, and command-line data pipelines.
Cover of DuckDB in Action
liveBookmanning.com
Read DuckDB in Action on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of DuckDB in Action. The section links point to the parts of the book it draws on.