Been running on Horovod on a multi-node setup for a while now, just recently realized i forgot to use the wrapper on the optimizer, hvd.DistributedOptimizer . Started redoing some of my runs and i seem to get the ~same results for both loss and accuracy when.

Is there some functionality that is auto enabled if you don't wrap the optimizer?