Installation failed #3070

bulant · 2021-06-29T00:05:09Z

bulant
Jun 29, 2021

Hello all,
First of all, thank you for the great work.
I am having problems installing horovod with tensorflow and mpi support.
My OS is Ubuntu 18.04.
I have conda 4.10.1
I am doing the following steps:

conda create -n ccta-tensorflow-cpu python=3.9
conda activate ccta-tensorflow-cpu
conda install -c conda-forge gxx_linux-64 gcc_linux-64 libgcc openmpi
conda install -c conda-forge tensorflow
HOROVOD_WITH_MPI=1 HOROVOD_WITH_TENSORFLOW=1 pip install --no-cache-dir horovod[tensorflow,keras]

The prompt output follows

...

Building wheels for collected packages: horovod
  Building wheel for horovod (setup.py) ... done
  Created wheel for horovod: filename=horovod-0.22.1-cp39-cp39-linux_x86_64.whl size=12503523 sha256=b70ef0a097500e80f9a86a6f249be19e86c6f853cb0640dab4079f80c670f26b
  Stored in directory: /tmp/pip-ephem-wheel-cache-w22y1rd3/wheels/96/e5/03/25cc44fc95d555a5858399c15c6d2f8325bb12116b26a4e0dc
Successfully built horovod
Installing collected packages: six, typing-extensions, pyyaml, psutil, cloudpickle, keras, horovod
  Attempting uninstall: six
    Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:
      Successfully uninstalled six-1.16.0
  Attempting uninstall: typing-extensions
    Found existing installation: typing-extensions 3.10.0.0
    Uninstalling typing-extensions-3.10.0.0:
      Successfully uninstalled typing-extensions-3.10.0.0
Successfully installed cloudpickle-1.6.0 horovod-0.22.1 keras-2.4.3 psutil-5.8.0 pyyaml-5.4.1 six-1.15.0 typing-extensions-3.7.4.3

Although no error are shown, hen checking installation, the tensorflow and MPI support are not available:

$ horovodrun --check-build
Horovod v0.22.1:

Available Frameworks:
    [ ] TensorFlow
    [X] PyTorch
    [ ] MXNet

Available Controllers:
    [ ] MPI
    [ ] Gloo

Available Tensor Operations:
    [ ] NCCL
    [ ] DDL
    [ ] CCL
    [ ] MPI
    [ ] Gloo

Also, trying to import horovod.tensorflow from python prompts the following error

$ python
Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
>>> import horovod.tensorflow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/carlos/develop/miniconda3/envs/ccta-tensorflow-cpu/lib/python3.9/site-packages/horovod/tensorflow/__init__.py", line 26, in <module>
    from horovod.tensorflow import elastic
  File "/home/carlos/develop/miniconda3/envs/ccta-tensorflow-cpu/lib/python3.9/site-packages/horovod/tensorflow/elastic.py", line 24, in <module>
    from horovod.tensorflow.functions import broadcast_object, broadcast_object_fn, broadcast_variables
  File "/home/carlos/develop/miniconda3/envs/ccta-tensorflow-cpu/lib/python3.9/site-packages/horovod/tensorflow/functions.py", line 24, in <module>
    from horovod.tensorflow.mpi_ops import allgather, broadcast
  File "/home/carlos/develop/miniconda3/envs/ccta-tensorflow-cpu/lib/python3.9/site-packages/horovod/tensorflow/mpi_ops.py", line 50, in <module>
    raise e
  File "/home/carlos/develop/miniconda3/envs/ccta-tensorflow-cpu/lib/python3.9/site-packages/horovod/tensorflow/mpi_ops.py", line 47, in <module>
    MPI_LIB = _load_library('mpi_lib' + get_ext_suffix())
  File "/home/carlos/develop/miniconda3/envs/ccta-tensorflow-cpu/lib/python3.9/site-packages/horovod/tensorflow/mpi_ops.py", line 42, in _load_library
    library = load_library.load_op_library(filename)
  File "/home/carlos/develop/miniconda3/envs/ccta-tensorflow-cpu/lib/python3.9/site-packages/tensorflow/python/framework/load_library.py", line 57, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/carlos/develop/miniconda3/envs/ccta-tensorflow-cpu/lib/python3.9/site-packages/horovod/tensorflow/mpi_lib.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZN10tensorflow6StatusC1ENS_5error4CodeEN4absl12lts_2021032411string_viewEOSt6vectorINS_10StackFrameESaIS7_EE
>>>

I really have no idea what is going on, there are no errors during installation, but somehow the installation process is unable to provide the tensorflow support.
I hope someone can help me with this.

Best regards Carlos.

maxhgerlach · 2021-07-14T16:55:06Z

maxhgerlach
Jul 14, 2021
Collaborator

Hi @bulant,

did you manage to sort out your build problem?

Installing TensorFlow via conda rather than pip might be problematic, not sure.

If you add a -v for verbose mode to your pip command line, you will get more log messages from the configuration and build process. That might give you an idea about what could be going wrong.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Installation failed #3070

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Installation failed #3070

Uh oh!

Uh oh!

bulant Jun 29, 2021

Replies: 1 comment

Uh oh!

maxhgerlach Jul 14, 2021 Collaborator

bulant
Jun 29, 2021

maxhgerlach
Jul 14, 2021
Collaborator