Available Tensor Operations do not support NCCL. #3104

powermano · 2021-08-13T07:27:02Z

powermano
Aug 13, 2021

I following the doc(https://github.com/horovod/horovod/blob/master/docs/gpus.rst).
Firstly install nccl:
The nccl_test:
make;

Compiling  all_reduce.cu                       > ../build/all_reduce.o
Compiling  common.cu                           > ../build/common.o
Linking  ../build/all_reduce.o               > ../build/all_reduce_perf
Compiling  all_gather.cu                       > ../build/all_gather.o
Linking  ../build/all_gather.o               > ../build/all_gather_perf
Compiling  broadcast.cu                        > ../build/broadcast.o
Linking  ../build/broadcast.o                > ../build/broadcast_perf
Compiling  reduce_scatter.cu                   > ../build/reduce_scatter.o
Linking  ../build/reduce_scatter.o           > ../build/reduce_scatter_perf
Compiling  reduce.cu                           > ../build/reduce.o
Linking  ../build/reduce.o                   > ../build/reduce_perf
Compiling  alltoall.cu                         > ../build/alltoall.o
Linking  ../build/alltoall.o                 > ../build/alltoall_perf

test:

#       size         count    type   redop     time   algbw   busbw  error     time   algbw   busbw  error
#        (B)    (elements)                     (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
           8             2   float     sum     6.13    0.00    0.00  0e+00     0.69    0.01    0.00  0e+00
          16             4   float     sum     6.11    0.00    0.00  0e+00     0.69    0.02    0.00  0e+00
          32             8   float     sum     6.33    0.01    0.00  0e+00     0.68    0.05    0.00  0e+00
          64            16   float     sum     6.18    0.01    0.00  0e+00     0.68    0.09    0.00  0e+00
         128            32   float     sum     6.17    0.02    0.00  0e+00     0.69    0.19    0.00  0e+00
         256            64   float     sum     6.11    0.04    0.00  0e+00     0.68    0.37    0.00  0e+00
         512           128   float     sum     5.97    0.09    0.00  0e+00     0.68    0.75    0.00  0e+00
        1024           256   float     sum     6.35    0.16    0.00  0e+00     0.68    1.50    0.00  0e+00
        2048           512   float     sum     6.25    0.33    0.00  0e+00     0.68    2.99    0.00  0e+00
        4096          1024   float     sum     5.93    0.69    0.00  0e+00     0.68    5.99    0.00  0e+00
        8192          2048   float     sum     5.97    1.37    0.00  0e+00     0.69   11.95    0.00  0e+00
       16384          4096   float     sum     6.31    2.60    0.00  0e+00     0.68   23.92    0.00  0e+00
       32768          8192   float     sum     6.02    5.44    0.00  0e+00     0.68   48.04    0.00  0e+00
       65536         16384   float     sum     5.99   10.94    0.00  0e+00     0.68   96.04    0.00  0e+00
      131072         32768   float     sum     6.23   21.03    0.00  0e+00     0.68  192.33    0.00  0e+00
      262144         65536   float     sum     6.40   40.93    0.00  0e+00     0.68  386.07    0.00  0e+00
      524288        131072   float     sum     7.14   73.43    0.00  0e+00     0.51  1020.31    0.00  0e+00
     1048576        262144   float     sum     7.85  133.65    0.00  0e+00     0.62  1695.90    0.00  0e+00
     2097152        524288   float     sum    12.69  165.23    0.00  0e+00     0.66  3164.80    0.00  0e+00
     4194304       1048576   float     sum    21.92  191.36    0.00  0e+00     0.67  6302.01    0.00  0e+00
     8388608       2097152   float     sum    41.21  203.56    0.00  0e+00     0.70  12007.74    0.00  0e+00
    16777216       4194304   float     sum    79.68  210.55    0.00  0e+00     0.72  23301.69    0.00  0e+00
    33554432       8388608   float     sum    156.7  214.10    0.00  0e+00     0.99  33998.11    0.00  0e+00
    67108864      16777216   float     sum    309.3  216.99    0.00  0e+00     0.62  108537.71    0.00  0e+00
   134217728      33554432   float     sum    614.9  218.27    0.00  0e+00     0.66  204693.81    0.00  0e+00
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 0

Then $ HOROVOD_GPU_OPERATIONS=NCCL pip install --no-cache-dir horovod

Horovod v0.19.5:

Available Frameworks:
    [ ] TensorFlow
    [X] PyTorch
    [X] MXNet

Available Controllers:
    [X] MPI
    [X] Gloo

Available Tensor Operations:
    [ ] NCCL
    [ ] DDL
    [ ] CCL
    [X] MPI
    [X] Gloo

envs: ubuntu16.04
mxnet: 1.8

maxhgerlach · 2021-08-13T07:53:09Z

maxhgerlach
Aug 13, 2021
Collaborator

If you installed NCCL manually, you may need to pass it's installed location as an environment variable when pip installing Horovod, see the docs.

Edit: Also Horovod 0.19.5 is quite old, why don't you use the most recent release?

4 replies

powermano Aug 13, 2021
Author

After installing NCCL, I got a /usr/lib/x86_64-linux-gnu/libnccl.so.2.10.3. Should I export HOROVOD_NCCL_HOME=/usr/lib/x86_64-linux-gnu.

I used python 3.5.2, it automatically install Horovod 0.19.5. I think it is time to upgrade python version.

maxhgerlach Aug 13, 2021
Collaborator

If nccl is not picked up from a standard location, you might just need to run ldconfig... If you install Horovod via pip install -v, you will get more better logging.

Anyway, updating Python sounds good!

powermano Aug 16, 2021
Author

Thanks, using ldconfig and then pip install finally works. Horovod + nccl much more faster than Horovod, ^_^。

powermano Aug 16, 2021
Author

Mon Aug 16 06:52:41 2021[0]<stderr>:2021-08-16 06:52:41,238 Epoch[1] Batch[7800] Speed: 460.21 samples/sec,  
Mon Aug 16 06:54:32 2021[0]<stderr>:2021-08-16 06:54:32,437 Epoch[1] Batch[8000] Speed: 460.44 samples/sec,  
Mon Aug 16 06:56:23 2021[0]<stderr>:2021-08-16 06:56:23,737 Epoch[1] Batch[8200] Speed: 460.03 samples/sec,  
Mon Aug 16 06:58:14 2021[0]<stderr>:2021-08-16 06:58:14,930 Epoch[1] Batch[8400] Speed: 460.47 samples/sec,  
Mon Aug 16 07:00:08 2021[0]<stderr>:2021-08-16 07:00:08,187 Epoch[1] Batch[8600] Speed: 452.07 samples/sec,  
Mon Aug 16 07:06:20 2021[0]<stderr>:2021-08-16 07:06:20,276 Epoch[1] Batch[8800] Speed: 137.63 samples/sec,  
Mon Aug 16 07:09:09 2021[0]<stderr>:2021-08-16 07:09:09,768 Epoch[1] Batch[9000] Speed: 303.06 samples/sec,  
Mon Aug 16 07:10:54 2021[0]<stderr>:2021-08-16 07:10:54,750 Epoch[1] Batch[9200] Speed: 487.71 samples/sec,

But sometimes, training speed is not stable.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Available Tensor Operations do not support NCCL. #3104

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Available Tensor Operations do not support NCCL. #3104

Uh oh!

powermano Aug 13, 2021

Replies: 1 comment · 4 replies

Uh oh!

Uh oh!

maxhgerlach Aug 13, 2021 Collaborator

Uh oh!

powermano Aug 13, 2021 Author

Uh oh!

maxhgerlach Aug 13, 2021 Collaborator

Uh oh!

powermano Aug 16, 2021 Author

Uh oh!

powermano Aug 16, 2021 Author

powermano
Aug 13, 2021

Replies: 1 comment 4 replies

maxhgerlach
Aug 13, 2021
Collaborator

powermano Aug 13, 2021
Author

maxhgerlach Aug 13, 2021
Collaborator

powermano Aug 16, 2021
Author

powermano Aug 16, 2021
Author