liveBook · Manning

Logical date

As explained in Data Pipelines with Apache Airflow, Second Edition

A logical date is the date or time label associated with the data interval or temporal point represented by a DAG run.

Suppose a daily DAG downloads events. If every run writes to `/data/events.json`, each new run overwrites the previous result. Airflow supplies `logical_date` in the task execution context: a date or time identifying what the DAG run represents. Putting it in the output path gives each run a separate file. section 3.3.1

Name files by logical date

A BashOperator can write to `/data/events/{{ logical_date | ds }}.json`; a downstream PythonOperator can read that same path and write `/data/stats/{{ logical_date | ds }}.csv`. Airflow renders the Jinja expressions for the run, so both tasks use matching date-stamped paths. section 3.3.1

Interpret the label according to the schedule:

  • With a trigger-based schedule, `logical_date` identifies the scheduled point in time for which the DAG runs. The scheduler starts the run as soon as possible after that point; the label need not be the moment a task actually executes. section 3.3.5
  • With an interval-based schedule, `logical_date` generally points to the interval start. Use `data_interval_start` and `data_interval_end` when processing records within the interval; the run executes after that interval has passed. section 3.4.4
  • An EventsTimetable also supplies a logical date for each specified event time, but does not supply meaningful data-interval boundaries. Incremental processing then requires calculating the needed date offsets separately. section 3.5

A date-stamped filename preserves separate run outputs, but it does not make an API response a daily extract. An endpoint returning the latest events may include more than one day and may return different data when a past run is repeated. For reproducible daily processing, query an explicit data interval rather than relying on the filename. section 3.3.1 section 3.4.4

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Explains the label across trigger, interval and event schedules, then demonstrates templated paths; does not detail execution-context internals.
Examples
Uses daily event and statistics filenames, interval-based runs and specified public-holiday event times.
What it assumes
Starts with a scheduled DAG and introduces the task execution-context value through a daily extract.
What you can do after
Apply it via Airflow task context and Jinja templating to identify runs and keep their output files separate.
How it connects
Connects logical date to DAG scheduling, Jinja templating, data intervals, catchup and incremental processing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.