A scheduler is the workflow-orchestration component that watches active workflows, determines when they should run, and dispatches their tasks according to the workflow DAG’s dependency order. It commonly operates alongside a controller, while workers provide the compute resources that execute the tasks. section 9.2.1 section 9.2.2
A production scheduler fits into the orchestration system by:
- Watching submitted and active workflows and scheduling them when their configured time arrives. section 9.2.2
- Dispatching workflow tasks to backend workers rather than performing the task’s heavy computation itself. section 9.2.1 section 9.2.2
- Respecting dependencies declared in the workflow DAG, such as parsing data before augmentation, dataset construction, and training. section 9.2.1
- Using shared orchestration state while workers exchange task outputs through the object store. section 9.2.2
Dependency-ordered training workflow
A monthly training workflow can define parsing, augmentation, dataset building, and training tasks, then declare each dependency with a directed chain. After submission, the scheduler detects the workflow and dispatches each task in that order. section 9.2.1
Use the system design to trace how the scheduler and controller connect the web server, metadata database, worker group, and object store. section 9.2.2
