I have built Horovod from source on a DGX A100 machine with the following backends:

(conda/2021-06-28/base) ➜  ~ horovodrun --check-build
Horovod v0.22.1:
Available Frameworks:
    [X] TensorFlow
    [X] PyTorch
    [ ] MXNet
Available Controllers:
    [X] MPI
    [X] Gloo
Available Tensor Operations:
    [X] NCCL
    [ ] DDL
    [ ] CCL
    [X] MPI
    [X] Gloo

Users of this installation use the MPI controller with NCCL for the majority of their work. However, one user has had deadlock issues and suspects it is due to NCCL.

While it is straightforward to switch controllers to the Gloo controller at runtime via horovodrun --gloo ..., it is not clear from the documentation as to whether or not it is possible to selectively disable NCCL, MPI Tensor Operation backends at runtime. I suspect not--- Horovod built in this way will always use a blend of Tensor Operations on a single CPU and GPU, indicated by:

When using an MPI controller, MPI will be used ... if tensors are placed in host memory prior to the allreduce request.

Are there any relevant environment variables that control the controller and/or Tensor Operation backends at runtime, similar to the compile-time flags HOROVOD_GPU_OPERATIONS, HOROVOD_CPU_OPERATIONS ? I have seen some references to HOROVOD_CONTROLLER, but not in official documentation.

Is there a way to see what the equivalent/full command is for horovdrun --gloo, like for mpirun in https://horovod.readthedocs.io/en/stable/mpi_include.html ?