I am implementing some time series classification using horovod on multi-GPUs. I initially used a simple 4-layers convolutional neural network and tested training time on single and double GPUs with 4 CPUs. I have seen a overhead on two GPUs compared to a single. Then I tried increasing parameters using LSTM and got the same result. Then I tried to increase the data, still got the same issue. I am confused about whether multi-GPUs are not efficient for training or if I am doing something wrong in implementation or utilization.

Environment:

1. Framework: (TensorFlow)
2. Framework version: 2.2.0
3. Horovod version: v0.21.3
4. MPI version: (Open MPI) 2.1.1
5. CUDA version: 10.1, V10.1.243
6. NCCL version: 2.11.4
7. Python version: 3.6.9
8. CMake version: 3.10.2

Running Command
srun --qos=gpu --gres=gpu:1 -n 2 -c 4 -C rtx2080,ubuntu18 -w gpu41--pty bash
/usr/local/bin/horovodrun -np 2 -H localhost:2 /usr/bin/python3 horov.py