OK, so that's quite a bit more of a tech stack than the original post alluded to. I don't know what Pyxis does. Presumably, srun functions like mpirun in your case (for MPI, horovodrun is a wrapper around mpirun). It's been a while since I last worked with SLURM, so that might also be standard...

You could set an environment variable like HOROVOD_LOG_LEVEL=trace to get more verbose logging and help you understand what's going on.

Yep, I wasn't aware about this: https://stackoverflow.com/questions/15578009/difference-between-running-a-program-with-and-without-mpirun

Interesting. So basically what you're saying is that Slurm's srun functions like mpirun (and like a "high-quality implementation"), and so it "gives" Horovod whatever it needs to recognize the "world" and the number of nodes and GPUs?

In conclusion, "horovodrun" is redundant when using Slurm?

Output with tracing enabled:
slurm-16164.txt

Thanks for the logs. They confirm that you are running with MPI (and that there's not some other magic enabling Gloo).

horovodrun can make it more convenient to set some configuration flags and there might be extra error handling, but if you don't need any of the extra features (like elastic mode, which wouldn't work with MPI), you don't absolutely need it.

Can you explain what happens first, is it that Slurm starts MPI and then Horovod wraps around it, or is it that Horovod starts and uses a backend of MPI? Or is it simpler than that, and just MPI being provided to Horovod which uses it as a backend?

Either way, the MPI backend is the one which allows Horovod to know the number of nodes and GPUs and enable multi-node runs automatically without "horovodrun"? I ask because when not using Horovod (E.g., PyTorch DDP) everything should be explicitly provided, such as the number of nodes, GPUs, ranks and so on (and also advised to be used with "torchrun"), even though it still uses a MPI backend. This is still "magic" to me when it come to Horovod.

BTW, up to this moment I wasn't able to run multi-node tasks with PT DDP and Slurm SBATCH, only with Horovod. But that another topic I'm trying to understand.