TF_CONFIG is an environment variable that identifies a TensorFlow distributed-training group. It gives each process the addresses of all workers and its own role within that group. section 4.2.3
A TF_CONFIG value describes two coordination components:
- cluster lists worker endpoints, such as IP addresses and ports, for the TensorFlow servers participating in training. section 4.2.3
- task identifies the current process with a role type and zero-based index; each worker receives a different task value. section 4.2.4
- The value is serialized into the TF_CONFIG environment variable before the training process starts. section 4.2.3
Two-worker configuration
A two-worker group can list 192.168.4.53:12345 and 192.168.4.55:23456 under cluster.worker. The process on the first worker identifies itself as {"type": "worker", "index": 0}; the other process uses its corresponding index. section 4.2.3
After TF_CONFIG is available, MultiWorkerMirroredStrategy uses the worker-group description for synchronous data-parallel training. The training service, rather than the data scientist, should usually generate addresses and task indexes when resources are dynamic. section 4.2.3 section 4.2.4
# Step 1: define 'TF_CONFIG' environment variable to describe
# the training group and the role for the process.
# The worker array defines the IP addresses and ports of
# all the TensorFlow servers used in this training.
tf_config = {
'cluster': {
'worker': ['192.168.4.53:12345', '192.168.4.55:23456']
},
# A 'task' provides information of the current task and is
# different for each worker. It specifies the 'type' and
# 'index' of that worker.
'task': {'type': 'worker', 'index': 0}
}
os.environ['TF_CONFIG'] = json.dumps(tf_config)
# Step 2: define distributed training strategy,
# the MultiWorkerMirroredStrategy takes
# care of the synchronous data parallel distributed training.
strategy = tf.distribute.MultiWorkerMirroredStrategy()
global_batch_size = per_worker_batch_size * num_workers
multi_worker_dataset = mnist.mnist_dataset(global_batch_size)
# Step 3: start the distributed training.
with strategy.scope():
# Model building/compiling need to be within 'strategy.scope()'.
multi_worker_model = mnist.build_and_compile_cnn_model()
multi_worker_model.fit(multi_worker_dataset,
epochs=3, steps_per_epoch=70)
Shows the complete setup sequence: construct TF_CONFIG, serialize it, create MultiWorkerMirroredStrategy, and derive the global batch size. section 4.2.3
In Kubernetes, a job tracker can collect worker-pod IPs or DNS names and broadcast a generated TF_CONFIG value to every pod, making cluster membership part of job launch. section 4.3.3
