liveBook · Manning

Master process

As explained in Designing Deep Learning Systems: A guide for software engineers

The designated coordinating process in a distributed training group, responsible for functions such as worker registration or providing initial model state.

A master process coordinates a distributed training group. During initialization, every process registers with it; one process takes the master role while the others act as workers, and registration waits until the group is complete. section 4.2.3

The master role supports several operational responsibilities:

  • Uses a globally unique rank, with rank 0 identifying the master, alongside world_size and the master's address and port. section 4.2.3
  • Provides the coordination point for distributed execution; in Kubernetes, a master-worker pod receives a dedicated service for worker connectivity. section 4.3.2
  • Acts as the reporting and persistence authority: inspect the master worker for synchronized progress, and save models or checkpoints only from the master. section 4.5

Three-worker Kubernetes training group

Submitting PARALLEL_INSTANCES=3 creates three training pods: one master worker and two normal workers. Kubernetes also creates a master service, while training status can be checked through the service API or the master worker. section 4.3.2

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes
Defines registration fields and operational responsibilities, but defers framework-specific communication internals to distributed-training libraries.
Examples
Uses PyTorch rank-0 initialization, a three-pod Kubernetes group, a master service, and master-only model or checkpoint persistence.
What it assumes
Assumes distributed training processes and workers, then defines ranks, group size, and master addressing for registration.
What you can do after
Operate distributed training by configuring master discovery, checking master-reported progress, and limiting saved artifacts to the master.
How it connects
Connects distributed training, data parallelism, worker processes, gradient aggregation, Kubernetes services, and Horovod.
Cover of Designing Deep Learning Systems: A guide for software engineers
liveBookmanning.com
Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.