Installing horovod pytorch with GPU support error #3408

Snoeprol · 2022-02-14T09:24:10Z

Snoeprol
Feb 14, 2022

I am trying to compile Horovod with the following command:

HOROVOD_NCCL_HOME=/sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/ HOROVOD_GPU_OPERATIONS=NCCL pip install --no-cache-dir horovod[pytorch]==0.21.0

I already added the environment variables

HOROVOD_WITH_PYTORCH=1 HOROVOD_WITHOUT_PYTORCH=0 HOROVOD_WITH_MXNET=0 HOROVOD_WITHOUT_MXNET=1 HOROVOD_WITH_TENSORFLOW=0 HOROVOD_WITHOUT_TENSORFLOW=1

And also

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/lib

I get the following error:

(/home/rooijm/Thesis/grace/env-tf1.15) rooijm@login4:/sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/lib$ HOROVOD_NCCL_HOME=/sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/ HOROVOD_GPU_OPERATIONS=NCCL pip install --no-cache-dir horovod[pytorch]==0.21.0
Defaulting to user installation because normal site-packages is not writeable
Collecting horovod[pytorch]==0.21.0
  Downloading horovod-0.21.0.tar.gz (3.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 142.0 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: cloudpickle in /sara/eb/AVX2/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/site-packages (from horovod[pytorch]==0.21.0) (1.6.0)
Requirement already satisfied: psutil in /sara/eb/AVX2/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/site-packages (from horovod[pytorch]==0.21.0) (5.8.0)
Requirement already satisfied: pyyaml in /sara/eb/AVX2/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/site-packages (from horovod[pytorch]==0.21.0) (5.4.1)
Requirement already satisfied: cffi>=1.4.0 in /sara/eb/AVX2/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/site-packages (from horovod[pytorch]==0.21.0) (1.14.5)
Requirement already satisfied: torch in /home/rooijm/.local/lib/python3.8/site-packages (from horovod[pytorch]==0.21.0) (1.10.1)
Requirement already satisfied: pycparser in /sara/eb/AVX2/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/site-packages (from cffi>=1.4.0->horovod[pytorch]==0.21.0) (2.20)
Requirement already satisfied: typing-extensions in /sara/eb/AVX2/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/site-packages (from torch->horovod[pytorch]==0.21.0) (3.7.4.3)
Building wheels for collected packages: horovod
  Building wheel for horovod (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [232 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.8
      creating build/lib.linux-x86_64-3.8/horovod
      copying horovod/__init__.py -> build/lib.linux-x86_64-3.8/horovod
      creating build/lib.linux-x86_64-3.8/horovod/_keras
      copying horovod/_keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/_keras
      copying horovod/_keras/callbacks.py -> build/lib.linux-x86_64-3.8/horovod/_keras
      copying horovod/_keras/elastic.py -> build/lib.linux-x86_64-3.8/horovod/_keras
      creating build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/__init__.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/basics.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/elastic.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/exceptions.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/util.py -> build/lib.linux-x86_64-3.8/horovod/common
      creating build/lib.linux-x86_64-3.8/horovod/keras
      copying horovod/keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/keras
      copying horovod/keras/callbacks.py -> build/lib.linux-x86_64-3.8/horovod/keras
      copying horovod/keras/elastic.py -> build/lib.linux-x86_64-3.8/horovod/keras
      creating build/lib.linux-x86_64-3.8/horovod/mxnet
      copying horovod/mxnet/__init__.py -> build/lib.linux-x86_64-3.8/horovod/mxnet
      copying horovod/mxnet/functions.py -> build/lib.linux-x86_64-3.8/horovod/mxnet
      copying horovod/mxnet/mpi_ops.py -> build/lib.linux-x86_64-3.8/horovod/mxnet
      creating build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/__init__.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/elastic.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/runner.py -> build/lib.linux-x86_64-3.8/horovod/ray
      creating build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/gloo_run.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/js_run.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/launch.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/mpi_run.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/run_task.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/task_fn.py -> build/lib.linux-x86_64-3.8/horovod/runner
      creating build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/conf.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/gloo_run.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/mpi_run.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/runner.py -> build/lib.linux-x86_64-3.8/horovod/spark
      creating build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/__init__.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/compression.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/elastic.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/functions.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/gradient_aggregation.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/gradient_aggregation_eager.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/mpi_ops.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/sync_batch_norm.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/util.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      creating build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/compression.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/functions.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/mpi_ops.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/optimizer.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/sync_batch_norm.py -> build/lib.linux-x86_64-3.8/horovod/torch
      creating build/lib.linux-x86_64-3.8/horovod/runner/common
      copying horovod/runner/common/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/common
      creating build/lib.linux-x86_64-3.8/horovod/runner/driver
      copying horovod/runner/driver/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/driver
      copying horovod/runner/driver/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/driver
      creating build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/constants.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/discovery.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/driver.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/registration.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/rendezvous.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/settings.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/worker.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      creating build/lib.linux-x86_64-3.8/horovod/runner/http
      copying horovod/runner/http/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/http
      copying horovod/runner/http/http_client.py -> build/lib.linux-x86_64-3.8/horovod/runner/http
      copying horovod/runner/http/http_server.py -> build/lib.linux-x86_64-3.8/horovod/runner/http
      creating build/lib.linux-x86_64-3.8/horovod/runner/task
      copying horovod/runner/task/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/task
      copying horovod/runner/task/task_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/task
      creating build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/cache.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/lsf.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/network.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/remote.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/threads.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      creating build/lib.linux-x86_64-3.8/horovod/runner/common/service
      copying horovod/runner/common/service/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/service
      copying horovod/runner/common/service/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/service
      copying horovod/runner/common/service/task_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/service
      creating build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/codec.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/config_parser.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/env.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/host_hash.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/hosts.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/network.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/safe_shell_exec.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/secret.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/settings.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/timeout.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/tiny_shell_exec.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      creating build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/_namedtuple_fix.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/backend.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/cache.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/constants.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/estimator.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/params.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/serialization.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/store.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/util.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      creating build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/host_discovery.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/job_id.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/mpirun_rsh.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/rendezvous.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/rsh.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      creating build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/bare.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/estimator.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/optimizer.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/remote.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/tensorflow.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/util.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      creating build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/gloo_exec_fn.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/mpirun_exec_fn.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/task_info.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/task_service.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      creating build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/estimator.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/remote.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/util.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      creating build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/callbacks.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/elastic.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      creating build/lib.linux-x86_64-3.8/horovod/torch/elastic
      copying horovod/torch/elastic/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch/elastic
      copying horovod/torch/elastic/sampler.py -> build/lib.linux-x86_64-3.8/horovod/torch/elastic
      copying horovod/torch/elastic/state.py -> build/lib.linux-x86_64-3.8/horovod/torch/elastic
      creating build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib
      copying horovod/torch/mpi_lib/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib
      creating build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib_impl
      copying horovod/torch/mpi_lib_impl/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib_impl
      running build_ext
      -- Could not find CCache. Consider installing CCache to speed up compilation.
      -- The CXX compiler identification is GNU 7.3.0
      -- Check for working CXX compiler: /home/rooijm/Thesis/grace/env-tf1.15/bin/x86_64-conda_cos6-linux-gnu-c++
      -- Check for working CXX compiler: /home/rooijm/Thesis/grace/env-tf1.15/bin/x86_64-conda_cos6-linux-gnu-c++ - works
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build architecture flags: -mf16c -mavx -mfma
      -- Using command /sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/bin/python
      -- Found MPI_CXX: /home/rooijm/Thesis/grace/env-tf1.15/lib/libmpicxx.so (found version "3.1")
      -- Found MPI: TRUE (found version "3.1")
      -- Found CUDA: /sw/arch/Debian10/EB_production/2021/software/CUDA/11.3.1 (found version "11.3")
      -- Linking against static NCCL library
      -- Found NCCL: /sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/include
      -- Determining NCCL version from the header file: /sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/include/nccl.h
      -- NCCL_MAJOR_VERSION: 2
      -- Found NCCL (include: /sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/include, library: /sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/lib/libnccl_static.a)
      -- The C compiler identification is GNU 7.3.0
      -- Check for working C compiler: /home/rooijm/Thesis/grace/env-tf1.15/bin/x86_64-conda_cos6-linux-gnu-cc
      -- Check for working C compiler: /home/rooijm/Thesis/grace/env-tf1.15/bin/x86_64-conda_cos6-linux-gnu-cc - works
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Found MPI_C: /home/rooijm/Thesis/grace/env-tf1.15/lib/libmpi.so (found version "3.1")
      -- Found MPI: TRUE (found version "3.1")
      -- MPI include path: /home/rooijm/Thesis/grace/env-tf1.15/include
      -- MPI libraries: /home/rooijm/Thesis/grace/env-tf1.15/lib/libmpicxx.so/home/rooijm/Thesis/grace/env-tf1.15/lib/libmpi.so
      -- Found Tensorflow: -L/home/rooijm/.local/lib/python3.8/site-packages/tensorflow;-l:libtensorflow_framework.so.2 (found suitable version "2.7.0", minimum required is "1.15.0")
      -- Found Pytorch: 1.10.1+cu102 (found suitable version "1.10.1+cu102", minimum required is "1.2.0")
      -- Found Mxnet: /home/rooijm/.local/lib/python3.8/site-packages/mxnet/libmxnet.so (found suitable version "1.9.0", minimum required is "1.4.0")
      CMake Error at horovod/mxnet/CMakeLists.txt:19 (message):
        Horovod build with GPU support was requested but this MXNet installation
        does not support CUDA.
      
      
      -- Configuring incomplete, errors occurred!
      See also "/scratch/pip-install-f9lb4spm/horovod_af9a93ca47d44092934e74fe3b96958d/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/scratch/pip-install-f9lb4spm/horovod_af9a93ca47d44092934e74fe3b96958d/setup.py", line 144, in <module>
          setup(name='horovod',
        File "/home/rooijm/.local/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/rooijm/.local/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
          _build_ext.run(self)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
          _build_ext.build_ext.run(self)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/command/build_ext.py", line 340, in run
          self.build_extensions()
        File "/scratch/pip-install-f9lb4spm/horovod_af9a93ca47d44092934e74fe3b96958d/setup.py", line 88, in build_extensions
          subprocess.check_call([cmake_bin, self.extensions[0].cmake_lists_dir] + cmake_args,
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/scratch/pip-install-f9lb4spm/horovod_af9a93ca47d44092934e74fe3b96958d', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELWITHDEBINFO=/scratch/pip-install-f9lb4spm/horovod_af9a93ca47d44092934e74fe3b96958d/build/lib.linux-x86_64-3.8', '-DPYTHON_EXECUTABLE:FILEPATH=/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/bin/python']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for horovod
  Running setup.py clean for horovod
Failed to build horovod
Installing collected packages: horovod
  Running setup.py install for horovod ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for horovod did not run successfully.
  │ exit code: 1
  ╰─> [236 lines of output]
      running install
      /home/rooijm/.local/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.8
      creating build/lib.linux-x86_64-3.8/horovod
      copying horovod/__init__.py -> build/lib.linux-x86_64-3.8/horovod
      creating build/lib.linux-x86_64-3.8/horovod/_keras
      copying horovod/_keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/_keras
      copying horovod/_keras/callbacks.py -> build/lib.linux-x86_64-3.8/horovod/_keras
      copying horovod/_keras/elastic.py -> build/lib.linux-x86_64-3.8/horovod/_keras
      creating build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/__init__.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/basics.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/elastic.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/exceptions.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/util.py -> build/lib.linux-x86_64-3.8/horovod/common
      creating build/lib.linux-x86_64-3.8/horovod/keras
      copying horovod/keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/keras
      copying horovod/keras/callbacks.py -> build/lib.linux-x86_64-3.8/horovod/keras
      copying horovod/keras/elastic.py -> build/lib.linux-x86_64-3.8/horovod/keras
      creating build/lib.linux-x86_64-3.8/horovod/mxnet
      copying horovod/mxnet/__init__.py -> build/lib.linux-x86_64-3.8/horovod/mxnet
      copying horovod/mxnet/functions.py -> build/lib.linux-x86_64-3.8/horovod/mxnet
      copying horovod/mxnet/mpi_ops.py -> build/lib.linux-x86_64-3.8/horovod/mxnet
      creating build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/__init__.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/elastic.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/runner.py -> build/lib.linux-x86_64-3.8/horovod/ray
      creating build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/gloo_run.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/js_run.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/launch.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/mpi_run.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/run_task.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/task_fn.py -> build/lib.linux-x86_64-3.8/horovod/runner
      creating build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/conf.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/gloo_run.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/mpi_run.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/runner.py -> build/lib.linux-x86_64-3.8/horovod/spark
      creating build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/__init__.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/compression.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/elastic.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/functions.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/gradient_aggregation.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/gradient_aggregation_eager.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/mpi_ops.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/sync_batch_norm.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/util.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      creating build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/compression.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/functions.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/mpi_ops.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/optimizer.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/sync_batch_norm.py -> build/lib.linux-x86_64-3.8/horovod/torch
      creating build/lib.linux-x86_64-3.8/horovod/runner/common
      copying horovod/runner/common/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/common
      creating build/lib.linux-x86_64-3.8/horovod/runner/driver
      copying horovod/runner/driver/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/driver
      copying horovod/runner/driver/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/driver
      creating build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/constants.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/discovery.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/driver.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/registration.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/rendezvous.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/settings.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/worker.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      creating build/lib.linux-x86_64-3.8/horovod/runner/http
      copying horovod/runner/http/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/http
      copying horovod/runner/http/http_client.py -> build/lib.linux-x86_64-3.8/horovod/runner/http
      copying horovod/runner/http/http_server.py -> build/lib.linux-x86_64-3.8/horovod/runner/http
      creating build/lib.linux-x86_64-3.8/horovod/runner/task
      copying horovod/runner/task/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/task
      copying horovod/runner/task/task_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/task
      creating build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/cache.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/lsf.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/network.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/remote.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/threads.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      creating build/lib.linux-x86_64-3.8/horovod/runner/common/service
      copying horovod/runner/common/service/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/service
      copying horovod/runner/common/service/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/service
      copying horovod/runner/common/service/task_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/service
      creating build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/codec.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/config_parser.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/env.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/host_hash.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/hosts.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/network.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/safe_shell_exec.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/secret.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/settings.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/timeout.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/tiny_shell_exec.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      creating build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/_namedtuple_fix.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/backend.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/cache.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/constants.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/estimator.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/params.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/serialization.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/store.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/util.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      creating build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/host_discovery.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/job_id.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/mpirun_rsh.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/rendezvous.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/rsh.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      creating build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/bare.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/estimator.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/optimizer.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/remote.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/tensorflow.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/util.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      creating build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/gloo_exec_fn.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/mpirun_exec_fn.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/task_info.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/task_service.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      creating build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/estimator.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/remote.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/util.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      creating build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/callbacks.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/elastic.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      creating build/lib.linux-x86_64-3.8/horovod/torch/elastic
      copying horovod/torch/elastic/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch/elastic
      copying horovod/torch/elastic/sampler.py -> build/lib.linux-x86_64-3.8/horovod/torch/elastic
      copying horovod/torch/elastic/state.py -> build/lib.linux-x86_64-3.8/horovod/torch/elastic
      creating build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib
      copying horovod/torch/mpi_lib/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib
      creating build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib_impl
      copying horovod/torch/mpi_lib_impl/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib_impl
      running build_ext
      -- Could not find CCache. Consider installing CCache to speed up compilation.
      -- The CXX compiler identification is GNU 7.3.0
      -- Check for working CXX compiler: /home/rooijm/Thesis/grace/env-tf1.15/bin/x86_64-conda_cos6-linux-gnu-c++
      -- Check for working CXX compiler: /home/rooijm/Thesis/grace/env-tf1.15/bin/x86_64-conda_cos6-linux-gnu-c++ - works
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build architecture flags: -mf16c -mavx -mfma
      -- Using command /sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/bin/python
      -- Found MPI_CXX: /home/rooijm/Thesis/grace/env-tf1.15/lib/libmpicxx.so (found version "3.1")
      -- Found MPI: TRUE (found version "3.1")
      -- Found CUDA: /sw/arch/Debian10/EB_production/2021/software/CUDA/11.3.1 (found version "11.3")
      -- Linking against static NCCL library
      -- Found NCCL: /sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/include
      -- Determining NCCL version from the header file: /sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/include/nccl.h
      -- NCCL_MAJOR_VERSION: 2
      -- Found NCCL (include: /sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/include, library: /sw/arch/Debian10/EB_production/2021/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/lib/libnccl_static.a)
      -- The C compiler identification is GNU 7.3.0
      -- Check for working C compiler: /home/rooijm/Thesis/grace/env-tf1.15/bin/x86_64-conda_cos6-linux-gnu-cc
      -- Check for working C compiler: /home/rooijm/Thesis/grace/env-tf1.15/bin/x86_64-conda_cos6-linux-gnu-cc - works
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Found MPI_C: /home/rooijm/Thesis/grace/env-tf1.15/lib/libmpi.so (found version "3.1")
      -- Found MPI: TRUE (found version "3.1")
      -- MPI include path: /home/rooijm/Thesis/grace/env-tf1.15/include
      -- MPI libraries: /home/rooijm/Thesis/grace/env-tf1.15/lib/libmpicxx.so/home/rooijm/Thesis/grace/env-tf1.15/lib/libmpi.so
      -- Found Tensorflow: -L/home/rooijm/.local/lib/python3.8/site-packages/tensorflow;-l:libtensorflow_framework.so.2 (found suitable version "2.7.0", minimum required is "1.15.0")
      -- Found Pytorch: 1.10.1+cu102 (found suitable version "1.10.1+cu102", minimum required is "1.2.0")
      -- Found Mxnet: /home/rooijm/.local/lib/python3.8/site-packages/mxnet/libmxnet.so (found suitable version "1.9.0", minimum required is "1.4.0")
      CMake Error at horovod/mxnet/CMakeLists.txt:19 (message):
        Horovod build with GPU support was requested but this MXNet installation
        does not support CUDA.
      
      
      -- Configuring incomplete, errors occurred!
      See also "/scratch/pip-install-f9lb4spm/horovod_af9a93ca47d44092934e74fe3b96958d/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/scratch/pip-install-f9lb4spm/horovod_af9a93ca47d44092934e74fe3b96958d/setup.py", line 144, in <module>
          setup(name='horovod',
        File "/home/rooijm/.local/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/rooijm/.local/lib/python3.8/site-packages/setuptools/command/install.py", line 68, in run
          return orig.install.run(self)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/command/install.py", line 545, in run
          self.run_command('build')
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/rooijm/.local/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
          _build_ext.run(self)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
          _build_ext.build_ext.run(self)
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/distutils/command/build_ext.py", line 340, in run
          self.build_extensions()
        File "/scratch/pip-install-f9lb4spm/horovod_af9a93ca47d44092934e74fe3b96958d/setup.py", line 88, in build_extensions
          subprocess.check_call([cmake_bin, self.extensions[0].cmake_lists_dir] + cmake_args,
        File "/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/lib/python3.8/subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/scratch/pip-install-f9lb4spm/horovod_af9a93ca47d44092934e74fe3b96958d', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELWITHDEBINFO=/scratch/pip-install-f9lb4spm/horovod_af9a93ca47d44092934e74fe3b96958d/build/lib.linux-x86_64-3.8', '-DPYTHON_EXECUTABLE:FILEPATH=/sw/arch/Debian10/EB_production/2021/software/Anaconda3/2021.05/bin/python']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> horovod

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

I don't understand why I get the error"

CMake Error at horovod/mxnet/CMakeLists.txt:19 (message): Horovod build with GPU support was requested but this MXNet installation does not support CUDA.

I am not even trying to install Horovod with mxnet support?

I have the following modules loaded on the cluster:

`Currently Loaded Modules:

2021
Anaconda3/2021.05
GCCcore/10.3.0
CUDA/11.3.1
zlib/1.2.11-GCCcore-10.3.0
numactl/2.0.14-GCCcore-10.3.0
UCX/1.10.0-GCCcore-10.3.0
GDRCopy/2.2-GCCcore-10.3.0
UCX-CUDA/1.10.0-GCCcore-10.3.0-CUDA-11.3.1
NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1`

My pip freeze gives the following

absl-py==1.0.0
aiohttp==3.8.1
aiosignal==1.2.0
alabaster @ file:///home/ktietz/src/ci/alabaster_1611921544520/work
anaconda-client==1.7.2
anaconda-navigator==2.0.3
anaconda-project @ file:///tmp/build/80754af9/anaconda-project_1610472525955/work
anyio @ file:///tmp/build/80754af9/anyio_1617783275907/work/dist
appdirs==1.4.4
argh==0.26.2
argon2-cffi @ file:///tmp/build/80754af9/argon2-cffi_1613037097816/work
asn1crypto @ file:///tmp/build/80754af9/asn1crypto_1596577642040/work
astroid @ file:///tmp/build/80754af9/astroid_1613500854201/work
astropy @ file:///tmp/build/80754af9/astropy_1617745353437/work
astunparse==1.6.3
async-generator @ file:///home/ktietz/src/ci/async_generator_1611927993394/work
async-timeout==4.0.2
atomicwrites==1.4.0
attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work
autopep8 @ file:///tmp/build/80754af9/autopep8_1615918855173/work
Babel @ file:///tmp/build/80754af9/babel_1607110387436/work
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
backports.functools-lru-cache @ file:///tmp/build/80754af9/backports.functools_lru_cache_1618170165463/work
backports.shutil-get-terminal-size @ file:///tmp/build/80754af9/backports.shutil_get_terminal_size_1608222128777/work
backports.tempfile @ file:///home/linux1/recipes/ci/backports.tempfile_1610991236607/work
backports.weakref==1.0.post1
beautifulsoup4 @ file:///home/linux1/recipes/ci/beautifulsoup4_1610988766420/work
bitarray @ file:///tmp/build/80754af9/bitarray_1620827551536/work
bkcharts==0.2
black==19.10b0
bleach @ file:///tmp/build/80754af9/bleach_1612211392645/work
bokeh @ file:///tmp/build/80754af9/bokeh_1620779595936/work
boto==2.49.0
Bottleneck==1.3.2
brotlipy==0.7.0
cachetools==4.2.4
certifi==2020.12.5
cffi @ file:///tmp/build/80754af9/cffi_1613246945912/work
chardet @ file:///tmp/build/80754af9/chardet_1607706746162/work
charset-normalizer==2.0.10
click @ file:///home/linux1/recipes/ci/click_1610990599742/work
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1598884132938/work
clyent==1.2.2
colorama @ file:///tmp/build/80754af9/colorama_1607707115595/work
commonmark==0.9.1
conda==4.10.1
conda-build==3.21.4
conda-content-trust @ file:///tmp/build/80754af9/conda-content-trust_1617045594566/work
conda-package-handling @ file:///tmp/build/80754af9/conda-package-handling_1618262148928/work
conda-repo-cli @ file:///tmp/build/80754af9/conda-repo-cli_1620168426516/work
conda-token @ file:///tmp/build/80754af9/conda-token_1620076980546/work
conda-verify==3.4.2
configparser==5.2.0
contextlib2==0.6.0.post1
cryptography @ file:///tmp/build/80754af9/cryptography_1616769286105/work
cycler==0.10.0
Cython @ file:///tmp/build/80754af9/cython_1618435160151/work
cytoolz==0.11.0
dask @ file:///tmp/build/80754af9/dask-core_1617390489108/work
decorator @ file:///tmp/build/80754af9/decorator_1617916966915/work
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
diff-match-patch @ file:///tmp/build/80754af9/diff-match-patch_1594828741838/work
dill==0.3.4
diskcache==5.4.0
distributed @ file:///tmp/build/80754af9/distributed_1620902833129/work
docker-pycreds==0.4.0
docutils @ file:///tmp/build/80754af9/docutils_1620827984873/work
entrypoints==0.3
et-xmlfile==1.0.1
fastcache==1.1.0
filelock @ file:///home/linux1/recipes/ci/filelock_1610993975404/work
flake8 @ file:///tmp/build/80754af9/flake8_1615834841867/work
Flask @ file:///home/ktietz/src/ci/flask_1611932660458/work
flatbuffers==2.0
frozenlist==1.3.0
fsspec==2022.1.0
future==0.18.2
gast==0.4.0
gevent @ file:///tmp/build/80754af9/gevent_1616770671827/work
gitdb==4.0.9
GitPython==3.1.26
glob2 @ file:///home/linux1/recipes/ci/glob2_1610991677669/work
gmpy2==2.0.8
google-auth==2.3.3
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
-e git+https://github.com/Snoeprol/PPTechniques.git@a524fc5463f776db4aca76bdda5204b44159bdda#egg=grace_dl
graphviz==0.8.4
greenlet @ file:///tmp/build/80754af9/greenlet_1611957705398/work
grpcio==1.43.0
h5py==2.10.0
HeapDict==1.0.1
html5lib @ file:///tmp/build/80754af9/html5lib_1593446221756/work
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
imageio @ file:///tmp/build/80754af9/imageio_1617700267927/work
imagesize @ file:///home/ktietz/src/ci/imagesize_1611921604382/work
importlib-metadata==4.10.1
iniconfig @ file:///home/linux1/recipes/ci/iniconfig_1610983019677/work
intervaltree @ file:///tmp/build/80754af9/intervaltree_1598376443606/work
ipykernel @ file:///tmp/build/80754af9/ipykernel_1596207638929/work/dist/ipykernel-5.3.4-py3-none-any.whl
ipython @ file:///tmp/build/80754af9/ipython_1617120885885/work
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
ipywidgets @ file:///tmp/build/80754af9/ipywidgets_1610481889018/work
isort @ file:///tmp/build/80754af9/isort_1616355431277/work
itsdangerous @ file:///home/ktietz/src/ci/itsdangerous_1611932585308/work
jdcal==1.4.1
jedi @ file:///tmp/build/80754af9/jedi_1606932564285/work
jeepney @ file:///tmp/build/80754af9/jeepney_1606148855031/work
Jinja2 @ file:///tmp/build/80754af9/jinja2_1612213139570/work
joblib @ file:///tmp/build/80754af9/joblib_1613502643832/work
json5==0.9.5
jsonschema @ file:///tmp/build/80754af9/jsonschema_1602607155483/work
jupyter==1.0.0
jupyter-client @ file:///tmp/build/80754af9/jupyter_client_1616770841739/work
jupyter-console @ file:///tmp/build/80754af9/jupyter_console_1616615302928/work
jupyter-core @ file:///tmp/build/80754af9/jupyter_core_1612213311222/work
jupyter-packaging @ file:///tmp/build/80754af9/jupyter-packaging_1613502826984/work
jupyter-server @ file:///tmp/build/80754af9/jupyter_server_1616083640759/work
jupyterlab @ file:///tmp/build/80754af9/jupyterlab_1619133235951/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab-server @ file:///tmp/build/80754af9/jupyterlab_server_1617134334258/work
jupyterlab-widgets @ file:///tmp/build/80754af9/jupyterlab_widgets_1609884341231/work
keras==2.7.0
Keras-Preprocessing==1.1.2
keyring @ file:///tmp/build/80754af9/keyring_1614616740399/work
kiwisolver @ file:///tmp/build/80754af9/kiwisolver_1612282420641/work
lazy-object-proxy @ file:///tmp/build/80754af9/lazy-object-proxy_1616526917483/work
libarchive-c @ file:///tmp/build/80754af9/python-libarchive-c_1617780486945/work
libclang==12.0.0
llvmlite==0.36.0
locket==0.2.1
lxml @ file:///tmp/build/80754af9/lxml_1616443220220/work
Markdown==3.3.6
MarkupSafe==1.1.1
matplotlib @ file:///tmp/build/80754af9/matplotlib-suite_1613407855456/work
mccabe==0.6.1
mistune==0.8.4
mkl-fft==1.3.0
mkl-random @ file:///tmp/build/80754af9/mkl_random_1618853849286/work
mkl-service==2.3.0
mock @ file:///tmp/build/80754af9/mock_1607622725907/work
more-itertools @ file:///tmp/build/80754af9/more-itertools_1613676688952/work
mpmath==1.2.1
msgpack @ file:///tmp/build/80754af9/msgpack-python_1612287151062/work
multidict==5.2.0
multipledispatch==0.6.0
mxnet==1.9.0
mypy-extensions==0.4.3
navigator-updater==0.2.1
nbclassic @ file:///tmp/build/80754af9/nbclassic_1616085367084/work
nbclient @ file:///tmp/build/80754af9/nbclient_1614364831625/work
nbconvert @ file:///tmp/build/80754af9/nbconvert_1601914830498/work
nbformat @ file:///tmp/build/80754af9/nbformat_1617383369282/work
nest-asyncio @ file:///tmp/build/80754af9/nest-asyncio_1613680548246/work
networkx @ file:///tmp/build/80754af9/networkx_1598376031484/work
nltk @ file:///tmp/build/80754af9/nltk_1618327084230/work
nose @ file:///tmp/build/80754af9/nose_1606773131901/work
notebook @ file:///tmp/build/80754af9/notebook_1616443462982/work
numba @ file:///tmp/build/80754af9/numba_1616774046117/work
numexpr @ file:///tmp/build/80754af9/numexpr_1618856167419/work
numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1618497241363/work
numpydoc @ file:///tmp/build/80754af9/numpydoc_1605117425582/work
nvidia-ml-py==11.515.0
oauthlib==3.1.1
olefile==0.46
openpyxl @ file:///tmp/build/80754af9/openpyxl_1615411699337/work
opt-einsum==3.3.0
packaging @ file:///tmp/build/80754af9/packaging_1611952188834/work
pandas==1.2.4
pandocfilters @ file:///tmp/build/80754af9/pandocfilters_1605120460739/work
parso==0.7.0
partd @ file:///tmp/build/80754af9/partd_1618000087440/work
path @ file:///tmp/build/80754af9/path_1614022220526/work
pathlib2 @ file:///tmp/build/80754af9/pathlib2_1607024983162/work
pathspec==0.7.0
pathtools==0.1.2
patsy==0.5.1
pep8==1.7.1
petastorm==0.11.3
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow @ file:///tmp/build/80754af9/pillow_1617383569452/work
pkginfo==1.7.0
pluggy @ file:///tmp/build/80754af9/pluggy_1615976321666/work
ply==3.11
prometheus-client @ file:///tmp/build/80754af9/prometheus_client_1618088486455/work
promise==2.3
prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1616415428029/work
protobuf==3.19.3
psutil @ file:///tmp/build/80754af9/psutil_1612298023621/work
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py @ file:///tmp/build/80754af9/py_1607971587848/work
py4j==0.10.9.2
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle @ file:///home/ktietz/src/ci_mi/pycodestyle_1612807597675/work
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pycurl==7.43.0.6
pyDeprecate==0.3.1
pydocstyle @ file:///tmp/build/80754af9/pydocstyle_1616182067796/work
pyerfa @ file:///tmp/build/80754af9/pyerfa_1619390903914/work
pyflakes @ file:///home/ktietz/src/ci_ipy2/pyflakes_1612551159640/work
Pygments @ file:///tmp/build/80754af9/pygments_1615143339740/work
pylint @ file:///tmp/build/80754af9/pylint_1617135829881/work
pyls-black @ file:///tmp/build/80754af9/pyls-black_1607553132291/work
pyls-spyder @ file:///tmp/build/80754af9/pyls-spyder_1613849700860/work
pyodbc===4.0.0-unsupported
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
pyrsistent @ file:///tmp/build/80754af9/pyrsistent_1600141720057/work
PySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/work
pyspark==3.2.0
pytest==6.2.3
python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/work
python-jsonrpc-server @ file:///tmp/build/80754af9/python-jsonrpc-server_1600278539111/work
python-language-server @ file:///tmp/build/80754af9/python-language-server_1607972495879/work
pytorch-lightning==1.5.9
pytz @ file:///tmp/build/80754af9/pytz_1612215392582/work
PyWavelets @ file:///tmp/build/80754af9/pywavelets_1601658317819/work
pyxdg @ file:///tmp/build/80754af9/pyxdg_1603822279816/work
PyYAML==5.4.1
pyzmq==20.0.0
QDarkStyle==2.8.1
QtAwesome @ file:///tmp/build/80754af9/qtawesome_1615991616277/work
qtconsole @ file:///tmp/build/80754af9/qtconsole_1616775094278/work
QtPy==1.9.0
regex @ file:///tmp/build/80754af9/regex_1617569202463/work
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-oauthlib==1.3.0
rich==11.0.0
rope @ file:///tmp/build/80754af9/rope_1602264064449/work
rsa==4.8
Rtree @ file:///tmp/build/80754af9/rtree_1618420845272/work
ruamel-yaml-conda @ file:///tmp/build/80754af9/ruamel_yaml_1616016699510/work
scalene==1.4.2
scikit-image==0.18.1
scikit-learn @ file:///tmp/build/80754af9/scikit-learn_1614446682169/work
scipy @ file:///tmp/build/80754af9/scipy_1618855647378/work
seaborn @ file:///tmp/build/80754af9/seaborn_1608578541026/work
SecretStorage @ file:///tmp/build/80754af9/secretstorage_1614022784285/work
Send2Trash @ file:///tmp/build/80754af9/send2trash_1607525499227/work
sentry-sdk==1.5.4
shortuuid==1.0.8
simplegeneric==0.8.1
singledispatch @ file:///tmp/build/80754af9/singledispatch_1614366001199/work
sip==4.19.13
six @ file:///tmp/build/80754af9/six_1605205327372/work
smmap==5.0.0
sniffio @ file:///tmp/build/80754af9/sniffio_1614030475067/work
snowballstemmer @ file:///tmp/build/80754af9/snowballstemmer_1611258885636/work
sortedcollections @ file:///tmp/build/80754af9/sortedcollections_1611172717284/work
sortedcontainers @ file:///tmp/build/80754af9/sortedcontainers_1606865132123/work
soupsieve @ file:///tmp/build/80754af9/soupsieve_1616183228191/work
Sphinx @ file:///tmp/build/80754af9/sphinx_1620777493457/work
sphinxcontrib-applehelp @ file:///home/ktietz/src/ci/sphinxcontrib-applehelp_1611920841464/work
sphinxcontrib-devhelp @ file:///home/ktietz/src/ci/sphinxcontrib-devhelp_1611920923094/work
sphinxcontrib-htmlhelp @ file:///home/ktietz/src/ci/sphinxcontrib-htmlhelp_1611920974801/work
sphinxcontrib-jsmath @ file:///home/ktietz/src/ci/sphinxcontrib-jsmath_1611920942228/work
sphinxcontrib-qthelp @ file:///home/ktietz/src/ci/sphinxcontrib-qthelp_1611921055322/work
sphinxcontrib-serializinghtml @ file:///home/ktietz/src/ci/sphinxcontrib-serializinghtml_1611920755253/work
sphinxcontrib-websupport @ file:///tmp/build/80754af9/sphinxcontrib-websupport_1597081412696/work
spyder @ file:///tmp/build/80754af9/spyder_1616775618138/work
spyder-kernels @ file:///tmp/build/80754af9/spyder-kernels_1614030590686/work
SQLAlchemy @ file:///tmp/build/80754af9/sqlalchemy_1620712430742/work
statsmodels @ file:///tmp/build/80754af9/statsmodels_1614023746358/work
subprocess32==3.5.4
sympy @ file:///tmp/build/80754af9/sympy_1618252284338/work
tables==3.6.1
tblib @ file:///tmp/build/80754af9/tblib_1597928476713/work
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.7.0
tensorflow-estimator==2.7.0
tensorflow-io-gcs-filesystem==0.23.1
termcolor==1.1.0
terminado==0.9.4
testpath @ file:///home/ktietz/src/ci/testpath_1611930608132/work
textdistance @ file:///tmp/build/80754af9/textdistance_1612461398012/work
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
three-merge @ file:///tmp/build/80754af9/three-merge_1607553261110/work
tifffile==2020.10.1
tinyec==0.4.0
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
toolz @ file:///home/linux1/recipes/ci/toolz_1610987900194/work
torch==1.10.1
torchmetrics==0.7.0
torchvision==0.11.2
tornado @ file:///tmp/build/80754af9/tornado_1606942300299/work
tqdm @ file:///tmp/build/80754af9/tqdm_1615925068909/work
traitlets @ file:///home/ktietz/src/ci/traitlets_1611929699868/work
typed-ast @ file:///tmp/build/80754af9/typed-ast_1610484547928/work
typing-extensions @ file:///home/ktietz/src/ci_mi/typing_extensions_1612808209620/work
ujson @ file:///tmp/build/80754af9/ujson_1611259522456/work
unicodecsv==0.14.1
urllib3 @ file:///tmp/build/80754af9/urllib3_1615837158687/work
wandb==0.12.9
watchdog @ file:///tmp/build/80754af9/watchdog_1612471027849/work
wcwidth @ file:///tmp/build/80754af9/wcwidth_1593447189090/work
webencodings==0.5.1
Werkzeug @ file:///home/ktietz/src/ci/werkzeug_1611932622770/work
widgetsnbextension==3.5.1
wrapt==1.12.1
wurlitzer @ file:///tmp/build/80754af9/wurlitzer_1617224664226/work
xlrd @ file:///tmp/build/80754af9/xlrd_1608072521494/work
XlsxWriter @ file:///tmp/build/80754af9/xlsxwriter_1617224712951/work
xlwt==1.3.0
xmltodict==0.12.0
yapf @ file:///tmp/build/80754af9/yapf_1615749224965/work
yarl==1.7.2
yaspin==2.1.0
zict==2.0.0
zipp @ file:///tmp/build/80754af9/zipp_1615904174917/work
zope.event==4.5.0
zope.interface @ file:///tmp/build/80754af9/zope.interface_1616357211867/work

Suggestions are appreciated!

maxhgerlach · 2022-03-03T13:21:30Z

maxhgerlach
Mar 3, 2022
Collaborator

horovod[pytorch]==0.21.0

Could you please try with the current release, Horovod 0.24

0 replies

xuehy · 2023-04-07T11:41:48Z

xuehy
Apr 7, 2023

add a CUDA_HOME env variable before the installation command solves the problem for me.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Installing horovod pytorch with GPU support error #3408

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Installing horovod pytorch with GPU support error #3408

Uh oh!

Uh oh!

Snoeprol Feb 14, 2022

Replies: 2 comments

Uh oh!

maxhgerlach Mar 3, 2022 Collaborator

Uh oh!

xuehy Apr 7, 2023

Snoeprol
Feb 14, 2022

maxhgerlach
Mar 3, 2022
Collaborator

xuehy
Apr 7, 2023