liveBook · Manning

Get_conn

As explained in Data Pipelines with Apache Airflow, Second Edition

get_conn is a hook method that establishes or returns a connection to an external system.

get_conn is the hook method that establishes or returns a connection to an external system. In a custom Airflow hook, it reads connection details from the metastore, builds a configured session, and returns connection data for higher-level methods. section 9.2.1

MovieLens API hook

A MovielensHook can read the movielens Airflow connection, default to HTTP on port 8081, attach its credentials to requests.Session, and return a Connection containing the session and base URL. A later get_ratings method can use that connection for paginated API requests.

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Explains configuration, defaults, authentication, returned objects, and caching, while deferring higher-level API operations to hook methods.
Examples
Uses a MovieLens API hook with metastore credentials, HTTP defaults, a requests session, and a paginated ratings method.
What it assumes
Assumes familiarity with Python classes, HTTP sessions, and Airflow hooks while teaching connection-store configuration from the implementation.
What you can do after
Build it from scratch by implementing configurable and optionally cached connections inside a custom Airflow hook.
How it connects
Connects custom hooks with BaseHook, Airflow connections, HTTP APIs, sessions, and pagination methods.
Cover of Data Pipelines with Apache Airflow, Second Edition
liveBookmanning.com
Read Data Pipelines with Apache Airflow, Second Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.