@chenghlee @jakirkham Do you happen to know who we could ask to review this?

Think @seibert wrote the original CUDA driver version detection logic.

@kkraus14 and @jaimergp may also have thoughts (Jaime in particular has gone down the CUDA Windows rabbit hole before).

Am happy to look as well. Though this may be something that occurs in the new year.

Instead of using the CUDA driver API to accomplish this, we could use the nvml library which doesn't have the same restrictions with regards to CUDA_VISIBLE_DEVICES.

Some example code:

from ctypes import *

libnvml = CDLL("libnvidia-ml.so.1")  # Will need to handle linux vs windows logic here
status = libnvml.nvmlInit_v2()  # Should check the result of status here to ensure it succeeded

cudaDriverVersion = c_int()
status = libnvml.nvmlSystemGetCudaDriverVersion_v2(byref(cudaDriverVersion))  # Should check the result of status here to ensure it succeeded

# cudaDriverVersion is now a ctypes c_int with the CUDA driver version

we could use the nvml library which doesn't have the same restrictions with regards to CUDA_VISIBLE_DEVICES.

The NVML library publishes new APIs with _vN suffix (e.g., nvmlInit_v2). The newly versioned APIs (symbols in libnvidia-ml.so) may not exist on the user side if they use old drivers.

nvmlInit_v2 was added in the CUDA 5.5 release of NVML so this should be fine. We can move to use nvmlInit if we want to be even more conservative (which is silly IMO).

nvmlSystemGetCudaDriverVersion_v2 was added in CUDA 10 release of NVML so maybe we should avoid using this. It looks like nvmlSystemGetCudaDriverVersion was added between CUDA 8 and CUDA 9 in driver 378 (NVIDIA/nvidia-settings@c9e5453) so that would be pretty good as well.

A potentially bad idea: it looks like you can call cuDriverGetVersion without calling cuInit first and it still returns the correct version without error. This is definitely relying on implementation details of the CUDA driver and could change without notice in the future.

Hey all! Thank you @XuehaiPan for continuing to push for change.

I'm trying to understand the correctness of this patch given that we plan to tag conda 23.1.0 January 16. Is this ready?

On a different notw, there is a larger discussion we could have: now that this virtual package is implemented as a plugin, we could release this in a separate conda-cuda-virtual-package (or similar) package, maybe co-maintained with folks from Nvidia? @jakirkham @kkraus14 Any opinions on this?

Is this ready?

@jezdez I think it's ready, but it's depending on the opinions of the code reviewers.

Let me summarize the changes in this PR:

1. Re-add the libcuda.so search path for WSL support.

2. Put the whole CUDA detection function in a separate subprocess:

   1. Main process: Start a new subprocess with the spawn start method.
   2. Subprocess: Detect the CUDA version in the subprocess and put the result in a SimpleQueue.
      • New change: unset CUDA_VISIBLE_DEVICES before initializing the CUDA driver context.
   3. Main process: receive the result from the subprocess and return the result to the function caller.

Rationale

1. Loading a .so file in the main process is not safe. If the driver library is corrupted, then the conda program always fails (Segmentation Fault). Putting the CUDA detector in a daemon process, if it fails, the caller can return None rather than crash.
2. Better process environment management. The CUDA driver library requires valid CUDA_VISIBLE_DEVICES. Using subprocesses, we can separate the environment rather than mess them together.
3. Early release of the CUDA library handle. We never release the .so library handle once we load it. If we load the CUDA library in the main process, it will cause the NVIDIA device and kernel module in use during the whole conda program lifetime. With subprocess, we can release the handle as soon as we get the result and exit the subprocess.

It feels really funky to unset the CUDA_VISIBLE_DEVICES environment variable in a spawned subprocess. Almost feels like this is conda overstepping its bounds.

We could use NVML and avoid having to mess with the environment variable and then decide if we want to use the subprocess to more nicely load and unload the library / isolate it from crashing the main conda process.

Almost feels like this is conda overstepping its bounds.

@kkraus14 I agree. I can open another PR to refactor with the NVML library.

One concern is the version handling and backward compatibility guarantee of the NVML library functions. Different NVML libraries ships different symbols. Newer versioned functions may have extra arguments, and the new and the old ones sometimes are not inter-replaceable.

That may require extra maintenance work. For example, check whether the library is new enough to have newer versioned functions with hasattr(lib, '<symbol_vN>'), otherwise fallback to _v(N-1). Then call the function with proper arguments (may be different between versions). Every time when a new NVIDIA driver is released, we will also need to update the version handling again.

#define NVML_INIT_FLAG_NO_GPUS      1   //!< Don't fail nvmlInit() when no GPUs are found
#define NVML_INIT_FLAG_NO_ATTACH    2   //!< Don't attach GPUs

nvmlReturn_t DECLDIR nvmlInit(void);
nvmlReturn_t DECLDIR nvmlInit_v2(void);
nvmlReturn_t DECLDIR nvmlInit_v3(unsigned int flags);
nvmlReturn_t DECLDIR nvmlInitWithFlags(unsigned int flags);
nvmlReturn_t DECLDIR nvmlShutdown(void);

nvmlReturn_t DECLDIR nvmlSystemGetCudaDriverVersion(int *cudaDriverVersion);
nvmlReturn_t DECLDIR nvmlSystemGetCudaDriverVersion_v2(int *cudaDriverVersion);

/***************************************************************************************************/
/** @defgroup nvmlInitializationAndCleanup Initialization and Cleanup
 * This chapter describes the methods that handle NVML initialization and cleanup.
 * It is the user's responsibility to call \ref nvmlInit_v2() before calling any other methods, and
 * nvmlShutdown() once NVML is no longer being used.
 *  @{
 */
/***************************************************************************************************/

#define NVML_INIT_FLAG_NO_GPUS      1   //!< Don't fail nvmlInit() when no GPUs are found
#define NVML_INIT_FLAG_NO_ATTACH    2   //!< Don't attach GPUs

/**
 * Initialize NVML, but don't initialize any GPUs yet.
 *
 * \note nvmlInit_v3 introduces a "flags" argument, that allows passing boolean values
 *       modifying the behaviour of nvmlInit().
 * \note In NVML 5.319 new nvmlInit_v2 has replaced nvmlInit"_v1" (default in NVML 4.304 and older) that
 *       did initialize all GPU devices in the system.
 *
 * This allows NVML to communicate with a GPU
 * when other GPUs in the system are unstable or in a bad state.  When using this API, GPUs are
 * discovered and initialized in nvmlDeviceGetHandleBy* functions instead.
 *
 * \note To contrast nvmlInit_v2 with nvmlInit"_v1", NVML 4.304 nvmlInit"_v1" will fail when any detected GPU is in
 *       a bad or unstable state.
 *
 * For all products.
 *
 * This method, should be called once before invoking any other methods in the library.
 * A reference count of the number of initializations is maintained.  Shutdown only occurs
 * when the reference count reaches zero.
 *
 * @return
 *         - \ref NVML_SUCCESS                   if NVML has been properly initialized
 *         - \ref NVML_ERROR_DRIVER_NOT_LOADED   if NVIDIA driver is not running
 *         - \ref NVML_ERROR_NO_PERMISSION       if NVML does not have permission to talk to the driver
 *         - \ref NVML_ERROR_UNKNOWN             on any unexpected error
 */
nvmlReturn_t DECLDIR nvmlInit_v2(void);

/**
 * nvmlInitWithFlags is a variant of nvmlInit(), that allows passing a set of boolean values
 *       modifying the behaviour of nvmlInit().
 *       Other than the "flags" parameter it is completely similar to \ref nvmlInit_v2.
 *
 * For all products.
 *
 * @param flags                                 behaviour modifier flags
 *
 * @return
 *         - \ref NVML_SUCCESS                   if NVML has been properly initialized
 *         - \ref NVML_ERROR_DRIVER_NOT_LOADED   if NVIDIA driver is not running
 *         - \ref NVML_ERROR_NO_PERMISSION       if NVML does not have permission to talk to the driver
 *         - \ref NVML_ERROR_UNKNOWN             on any unexpected error
 */
nvmlReturn_t DECLDIR nvmlInitWithFlags(unsigned int flags);

/**
 * Shut down NVML by releasing all GPU resources previously allocated with \ref nvmlInit_v2().
 *
 * For all products.
 *
 * This method should be called after NVML work is done, once for each call to \ref nvmlInit_v2()
 * A reference count of the number of initializations is maintained.  Shutdown only occurs
 * when the reference count reaches zero.  For backwards compatibility, no error is reported if
 * nvmlShutdown() is called more times than nvmlInit().
 *
 * @return
 *         - \ref NVML_SUCCESS                 if NVML has been properly shut down
 *         - \ref NVML_ERROR_UNINITIALIZED     if the library has not been successfully initialized
 *         - \ref NVML_ERROR_UNKNOWN           on any unexpected error
 */
nvmlReturn_t DECLDIR nvmlShutdown(void);

/** @} */

/**
 * Retrieves the version of the CUDA driver.
 *
 * For all products.
 *
 * The CUDA driver version returned will be retreived from the currently installed version of CUDA.
 * If the cuda library is not found, this function will return a known supported version number.
 *
 * @param cudaDriverVersion                    Reference in which to return the version identifier
 *
 * @return
 *         - \ref NVML_SUCCESS                 if \a cudaDriverVersion has been set
 *         - \ref NVML_ERROR_INVALID_ARGUMENT  if \a cudaDriverVersion is NULL
 */
nvmlReturn_t DECLDIR nvmlSystemGetCudaDriverVersion(int *cudaDriverVersion);

/**
 * Retrieves the version of the CUDA driver from the shared library.
 *
 * For all products.
 *
 * The returned CUDA driver version by calling cuDriverGetVersion()
 *
 * @param cudaDriverVersion                    Reference in which to return the version identifier
 *
 * @return
 *         - \ref NVML_SUCCESS                  if \a cudaDriverVersion has been set
 *         - \ref NVML_ERROR_INVALID_ARGUMENT   if \a cudaDriverVersion is NULL
 *         - \ref NVML_ERROR_LIBRARY_NOT_FOUND  if \a libcuda.so.1 or libcuda.dll is not found
 *         - \ref NVML_ERROR_FUNCTION_NOT_FOUND if \a cuDriverGetVersion() is not found in the shared library
 */
nvmlReturn_t DECLDIR nvmlSystemGetCudaDriverVersion_v2(int *cudaDriverVersion);

/**
 * Macros for converting the CUDA driver version number to Major and Minor version numbers.
 */
#define NVML_CUDA_DRIVER_VERSION_MAJOR(v) ((v)/1000)
#define NVML_CUDA_DRIVER_VERSION_MINOR(v) (((v)%1000)/10)

Given that none of the older functions are deprecated or removed we could just use them and hold off on any kind of versioning until needed.

See my comment here: #11667 (comment)

Driver 378, which was when nvmlSystemGetCudaDriverVersion was included, released ~6 years ago so I think it's pretty reasonable that we don't need to be able to detect the CUDA version on drivers older than that. Besides, the oldest version of the CUDA toolkit packaged as a conda package is 8.0 and its only packaged for Windows on the anaconda channel. The oldest version packaged on conda-forge is 9.2.

@XuehaiPan @kkraus14 Before I forget, do you feel like it's important to file a ticket for the refactoring to use the nvml library?

Alternatively, does anyone know how likely it would be for us to extract the cuda virtual package into a separate repo? Any takers from Nvidia?

It's probably a good idea to have an issue

We could look into a plugin, but that won't be for a while

Highest priority for me is getting new packages together ( conda-forge/staged-recipes#21382 ) and starting CUDA 12 builds

NVML should be the way to go (e.g. see here). Sorry to see/raise this late, but this feels a bit awkward to me 😅 Like Keith said, the needed NVML symbols have existed for very long time, it should be preferred for detecting the driver version without initializing the context.