I'm using Horovod Spark Estimators API for my workload. Then I upgraded the Horovod and Tensorflow versions from Horovod 0.19.1 and TF 2.2 to Horovod 0.24.0 and TF 2.8.0. One use case of my workload(LSTM) got a 50% performance drop and the left workloads all got performance gain.
And if I tested the use case without Horovod, the TF 2.8 shows better performance than TF 2.2. So, I'm curious about how can I debug this performance issue for Horovod on spark? Thanks