Caching stores computed, retrieved, or intermediate data so later work can reuse it instead of repeating the original operation. In Airflow workflows, this supports efficient reruns: individual tasks can restart from available outputs rather than recomputing every dependency. section 12.3.3
Cache a hook connection
A custom API hook can cache its authenticated session and base URL in protected instance variables. The first get_conn call reads connection details from the Airflow metastore and creates the session; later calls return the cached objects without repeating that lookup. section 9.2.1
Apply caching to data pipelines with these operational choices:
- Store intermediate outputs where workers can access them, allowing downstream tasks to rerun independently. section 12.3.3
- Retain intermediate data for a limited period when storage costs make keeping every large version impractical. section 12.3.3
- Keep the rawest ingested data available so processing can be repeated after code changes or failures. section 12.3.3
- If checking for existing outputs, account for cases where intermediate data must be deliberately rewritten. section 12.3.3
Figure 12.11 Storing intermediate data from tasks ensures that we can easily run each task independently. In this case, cloud storage (indicated by the bucket) is used to store intermediate results of the fetch/preprocess tasks.
The workflow diagram shows cloud storage holding intermediate results from fetch and preprocess tasks, making each task independently rerunnable. section 12.3.3
