liveBook · Manning

Operator

As explained in Designing Deep Learning Systems: A guide for software engineers

A workflow component that executes a task according to a defined operation implementation.

An operator is the executable implementation behind a workflow task. It lets a DAG declare what each task does while the orchestration system schedules, dispatches, runs, and monitors that work. section 9.3.1 section 9.2.2

Choose an operator according to the task and the amount of customization required:

  • Use predefined or community operators such as MySqlOperator, SimpleHttpOperator, Docker operator, and PostgresOperator to implement common tasks without writing task logic. section 9.3.1
  • Use PythonOperator to run a customized Python function. section 9.3.1
  • Use a default shell or Python operator, or build a custom operator when the orchestration system lacks the required task behavior. section 9.2.2

Operators in a data workflow

A daily Airflow DAG can use FileSensor to wait for data.csv, PythonOperator to transform data, and PostgresOperator to create a database table. Dependencies determine their execution sequence. section 9.3.1

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes
Explains predefined, customized, and custom operators, then traces worker execution and output storage without detailing operator internals.
Examples
Uses FileSensor, PythonOperator, PostgresOperator, MySqlOperator, SimpleHttpOperator, Docker operator, and shell commands as task implementations.
What you can do after
Apply it via Airflow to define workflow tasks with predefined operators, custom Python functions, or custom implementations.
How it connects
Connects operators with DAGs, task dependencies, workers, schedulers, sensors, object stores, and metadata databases.

Also explained in

Compare how 2 books explain operator →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.