Hi,
First a simple question, does pytorch_imagenet_resnet50 example has early stoppage enabled in the code? Where if the validation loss isn't decreasing, it will forcefully stop training and complete the job? Whats weird for me is, on signal node, I get to ~70% accuracy, but when training on 4 GPU nodes, the job stops when its at ~25% accuracy. See image

Second question. If above doesn't have early stoppage, then what can cause the job to disconnect?

• I am using Polyaxon + Mpirun + Horovod.
• Running the job on EKS Cluster

I know there are many variables involved, but any direction would be highly appreciated.