Another issue I met is that I am not able to get the same result from what I saw on chrome://tracing and the json file. I extract the time based on pid but the results are a bit off. Any helps there? Thanks in advance!

For example, I have this json file from HOROVOD_TIMELINE and an extract time script gets ts based pid. But the results I got are different. For pid=28 I get 815.850ms from python script and 818.350ms from chrome://tracing.

+

I use a simple script to record the elapsed time for allreduce op but I am getting different elapsed time from torch.cuda.Event(enable_timing=True) and the built-in timeline of horovod.

There can be some small extra delays from framework code. I don't know too much about PyTorch, but at least with graph-based TensorFlow code these things tend to get better in subsequent training steps. How large are the discrepancies you've seen, btw?

I've found Nvidia's Nsight profiling tools really helpful to understand where my models spend time. To make this easier with Horovod, I've contributed a PR #2723 that allows you to both see (1) timing ranges approximately corresponding to your torch timings and (2) all the detailed Horovod timeline information in the same profiling display.

How large are the discrepancies you've seen, btw?

@maxhgerlach Hi Max, thanks for the reply. The horovod records NEGOTIATE_ALLREDUCE for ~4949ms for the discrepancy is really large for the first pid, the rest difference is a bit smaller but still about ~30% difference between time recorded by torch and horovod timeline

@dexhunter, on your plot the extra latency seems to grow with the message size. Might this just be latency incurred by copying around the data?

@maxhgerlach Hi thanks for the reply. From my understanding, with larger message sizes, the time for executing the operation will also grow. The plot is based on the timeline json file, so the time moving around the time should also be included as seen in WAIT_FOR_DATA or maybe I should change the ylabel to execution time?