For an algorithm I have written I am trying to get out the communication time of the algorithm. I think it is interleaved with computation as stated in other threads, but what would be the fairest way to compare two algorithms? I have two tracing files: one using SGD, and one using my own method. My main question then would be: what part of the tracing file would grow (roughly) proportional to the network speed. Is it

1. WAIT_FOR_DATA
2. WAIT_FOR_OTHER_TENSOR_DATA
3. NCCL_ALLREDUCE
4. A combination of the operations

Then for ALLGATHER I have the same question.

1. WAIT_FOR_DATA
2. WAIT_FOR_OTHER_TENSOR_DATA
3. NCCL_ALLGATHER
4. ALLOCATE_OUTPUT

Or a combination of above?

I have read the timeline readme, and with the NCCL_ALLREDUCE it states :

NCCL_ALLREDUCE, MPI_ALLREDUCE, MPI_ALLGATHER, or MPI_BCAST indicate time taken to do the actual operation on GPU (or CPU) and highlights whether the operation was performed using NCCL or pure MPI.

Is this the only operation where data is being sent and received?

Thanks in advance,

Mario