Estimator.fit() throws following error:

 File "/setup/env/lib/python3.8/site-packages/horovod/spark/common/estimator.py", line 35, in fit
    return super(HorovodEstimator, self).fit(df, params)
  File "/opt/spark/python/lib/pyspark.zip/pyspark/ml/base.py", line 161, in fit
    return self._fit(dataset)
  File "/setup/env/lib/python3.8/site-packages/horovod/spark/common/estimator.py", line 80, in _fit
    return self._fit_on_prepared_data(
  File "/setup/env/lib/python3.8/site-packages/horovod/spark/torch/estimator.py", line 282, in _fit_on_prepared_data
    handle = backend.run(trainer,
  File "/setup/env/lib/python3.8/site-packages/horovod/spark/common/backend.py", line 84, in run
    return horovod.spark.run(fn, args=args, kwargs=kwargs,
  File "/setup/env/lib/python3.8/site-packages/horovod/spark/runner.py", line 291, in run
    _launch_job(use_mpi, use_gloo, settings, driver, env, stdout, stderr)
  File "/setup/env/lib/python3.8/site-packages/horovod/spark/runner.py", line 153, in _launch_job
    run_controller(use_gloo, lambda: gloo_run(settings, nics, driver, env, stdout, stderr),
  File "/setup/env/lib/python3.8/site-packages/horovod/runner/launch.py", line 706, in run_controller
    gloo_run()
  File "/setup/env/lib/python3.8/site-packages/horovod/spark/runner.py", line 153, in 
    run_controller(use_gloo, lambda: gloo_run(settings, nics, driver, env, stdout, stderr),
  File "/setup/env/lib/python3.8/site-packages/horovod/spark/gloo_run.py", line 67, in gloo_run
    launch_gloo(command, exec_command, settings, nics, {}, server_ip)
  File "/setup/env/lib/python3.8/site-packages/horovod/runner/gloo_run.py", line 268, in launch_gloo
    raise RuntimeError('Horovod detected that one or more processes exited with non-zero '
RuntimeError: Horovod detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was:
Process name: 1

But spark logs (stdout/stderr outputs), don't have any inserted trace print statements/logs to stdout/stderr.
Any clues on why this would happen ? Appreciate any help on this issue.

This behavior is not consistent. When there's an error in data-access, I could see all stdout/err logs being captured.