Hi @maxhgerlach, thanks for your reply!
I have some follow-up questions here.

1. I still see quite many number of NCCL allreduce operations invoked in the detailed GPU stream view (screenshot below). Doesn't it mean that there are multiple allreduce operations are perforemd by NCCL?
2. Can we enable the horovod timeline inside the Nsight Systems, not the Chrome tracing view?
3. What does NCCL allreduce operation actually do here? Does it send the DMA write request to DMA controller to write the gradient in GPU memory to host memory?

It's still quite confusing to me.. Hope you could kindly help for me to clarify it. Thanks!

I still see quite many number of NCCL allreduce operations invoked in the detailed GPU stream view (screenshot below). Doesn't it mean that there are multiple allreduce operations are perforemd by NCCL?

Yes, but not one per tensor, so fusion is working. Are some of those for different data types maybe? Or do you have some non-allreduce operations in there as well?

You could also tune knobs like HOROVOD_FUSION_THRESHOLD and HOROVOD_CYCLE_TIME to influence the tensor fusion behavior if you feel the need. You could also try the Autotune mechanism, but I personally haven't ever.

Can we enable the horovod timeline inside the Nsight Systems, not the Chrome tracing view?

Yes, you will also get equivalent data in Nsight Systems if you enable writing the tracings that you can view in Chrome. See this PR description for some notes #2723 (has been merged and released for a while now, but I don't think there's better documentation yet...).

What does NCCL allreduce operation actually do here? Does it send the DMA write request to DMA controller to write the gradient in GPU memory to host memory?

That's very dependent on your GPU, mainboard, and networking. If you sign up with Nvidia, you could get access to this recent talk which describes what could be going on under the hood:
https://www.nvidia.com/en-us/on-demand/session/gtcspring22-s41784/

Thanks @maxhgerlach, the Nvidia talk is very informative.

Speaking of the enabling horovod timeline into the Nsight systems, since the PR is already merged and released, I should be able to see it in the Nsight Systems when I use the latest horovod, isn't it? I generated the above Nsight Systems profile using the latest horovod docker container, but I don't see the timeline in it. Is there anything I need to do to see it?

FYI, this is the command I use:
horovodrun -np 2 -H localhost:1,13.250.102.43:1 -p 12345 --verbose nsys profile -t cuda,osrt,nvtx,cudnn,cublas -f true -o ./baseline -w true python /root/byteps/bytescheduler/examples/pytorch_horovod_benchmark.py --num-iters 2 --model resnet50 --num-classes 10000
Thanks!

Hi @ihchoi12, to get the detailed timeline-like annotations you currently need to enable writing the timeline data to a tracing file, too. For example, set an environment variable like HOROVOD_TIMELINE=/path/to/timeline.json. Gathering that data adds some runtime overhead, that's why it's optional. Granted, it's a bit confusing that it's coupled to writing that file -- even if you don't want to look at it, but only want to use the profiler. There's always room for improvement. :)