thanks @maxhgerlach! Also in Elastic scheme of things, can you please shed some light on what happens when a rank-0 process(node) dies. As per my understanding, the training should continue so long as min_np value is satisfied. But rank-0 process is also supposed to save model artifacts, tensorboard logs etc. which is essential. So, does Horovod make any other process as rank-0 ?

In my testing of this scenario, my cluster just stops.

2022-05-23 10:33:26 - [2]<stderr>:[2022-05-23 10:33:26.681578: E /tmp/pip-install-6gw2lcfc/horovod_130c09307b8b4f0e97b7e4670611d00e/horovod/common/operations.cc:654] [2]: Horovod background loop uncaught exception: [/tmp/pip-install-6gw2lcfc/horovod_130c09307b8b4f0e97b7e4670611d00e/third_party/compatible_gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 30000ms for recv operation to complete 2022-05-23 10:33:28 - [2]<stderr>:2022-05-23 10:33:28.002542: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at mpi_ops.cc:804 : Unknown: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message

Thanks again !