A queue is an ordered data structure from which work items are retrieved for processing. In workflow systems, a message queue carries near-real-time messages that represent individual events, such as bank transactions or completed data uploads. section 7.0 section 7.3
Airflow can use queue messages as signals to start a DAG. A provider-specific trigger watches a broker and topic, optionally filters messages with an apply_function, and exposes the result through an asset used in the DAG schedule. section 7.3
Queue-backed workflow triggers support several practical patterns:
- Replace polling for a completion file with a message stating that a supermarket has finished uploading data. section 7.3
- Filter messages so only events containing the required value trigger a workflow; returning True accepts every message. section 7.3
- Route CeleryExecutor tasks to separate workers and queues so teams can use different dependency images. section 17.2.8
Kafka message triggers a DAG
A MessageQueueTrigger watches kafka://kafka:9092/events and applies a custom filter. An AssetWatcher attaches that trigger to kafka_queue_asset; a DAG scheduled on the asset then runs copy_to_raw_supermarket_1 >> process_supermarket_1 >> create_metrics.
