-
|
Using 2 VMs with 1 GPU each to do distributed training using Horovod. This is what I have for building and compiling my model: When I pass in True for horovod, my code breaks starts Epoch 1, then hangs for a while (no output), and then crashes with the following error. When I pass in False for horovod, my code works on 2 VMs. My hvd.init() and model.fit() are both in a script called horovod_training.py. Is this okay to do? Any help is appreciated! |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 3 replies
-
|
Do the Keras examples from the Horovod repo work for you? https://github.com/horovod/horovod/tree/master/examples/keras Those examples use |
Beta Was this translation helpful? Give feedback.
-
|
I tested it with https://github.com/horovod/horovod/blob/master/examples/tensorflow2/tensorflow2_keras_mnist.py and it worked. This code was using tf.optimizers.Adam(scaled_lr). |
Beta Was this translation helpful? Give feedback.
-
|
Let's move this to Discussion as the problem is probably specific to your script. |
Beta Was this translation helpful? Give feedback.
-
|
Ok would you like to see some specific part of my code? |
Beta Was this translation helpful? Give feedback.
-
|
@maxhgerlach |
Beta Was this translation helpful? Give feedback.
-
|
I don't know your code and I wouldn't have time to study it. In your screenshot it looks like different ranks try to submit different tensors to the allreduce at the same time, thus blocking each other. That won't work. |
Beta Was this translation helpful? Give feedback.
-
|
So how do you recommend I go about debugging this? |
Beta Was this translation helpful? Give feedback.
-
|
So I think I got it working. I was using the tensorboard callback and only adding that to my chief worker, this was causing some rank issues. Now every worker gets the tensorboard callback, and the horovod training seems to run just fine! |
Beta Was this translation helpful? Give feedback.
So I think I got it working. I was using the tensorboard callback and only adding that to my chief worker, this was causing some rank issues. Now every worker gets the tensorboard callback, and the horovod training seems to run just fine!