ue Aug 31 06:03:33 2021[0]<stderr>:[2021-08-31 06:03:33.894779: W horovod/common/stall_inspector.cc:105] One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock. 
Tue Aug 31 06:03:33 2021[0]<stderr>:Stalled ranks:
Tue Aug 31 06:03:33 2021[0]<stderr>:1: [horovod_allreduce.h0_x_xml_score.xml_score.weight]
Tue Aug 31 06:03:33 2021[0]<stderr>:2: [horovod_allreduce.h0_x_xml_score.xml_score.weight]
Tue Aug 31 06:04:33 2021[0]<stderr>:[2021-08-31 06:04:33.897124: W horovod/common/stall_inspector.cc:105] One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock. 
Tue Aug 31 06:04:33 2021[0]<stderr>:Stalled ranks:
Tue Aug 31 06:04:33 2021[0]<stderr>:0: [horovod_allreduce.x_total_emb_feat, horovod_allreduce.x_total_label]
Tue Aug 31 06:04:33 2021[0]<stderr>:1: [horovod_allreduce.h0_x_xml_score.xml_score.weight]
Tue Aug 31 06:04:33 2021[0]<stderr>:2: [horovod_allreduce.h0_x_xml_score.xml_score.weight]
^CProcess 1 exit with status code 143.