Python.NET & TensorFlow & CUDA: Could not load dynamic library 'cublas64_11.dll'

Environment

• Pythonnet version:
• Python version:
• Operating System:
• .NET Runtime:
• GeForce RTX 3080 (computeCapability: 8.6)
• Windows 10 20H2
• Net Framework 4.8 (Compiled with Visual Studio 2019)
• Python 3.7 (64-bit)
• Nvidia CUDA 11.
• Tensforflow-gpu 2.4.1
• Tf-Agents 0.7.1
• pythonnet 3.0.0-preview2021-04-03

Details

I am current working on using Python.NET to build C# environments for interaction TensorFlow Agents and am receiving a TensorFlow error attempting to load Cuda DLLs.

When I run pure python examples Tensor flow loads the CUDA DLLs without issue:

2021-04-19 03:22:41.062449: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-19 03:22:41.062943: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-19 03:22:41.063347: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-19 03:22:41.063709: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-04-19 03:22:41.064088: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-04-19 03:22:41.064455: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-04-19 03:22:41.064832: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-04-19 03:22:41.065202: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll

However, when I run an environment that uses a Python environment that is essentially a wrapper for an environment written in C# using Python.Net is recieve errors the Cuda DLLs were not found:

2021-04-19 03:15:14.884746: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2021-04-19 03:15:14.885031: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2021-04-19 03:15:14.885281: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2021-04-19 03:15:14.885586: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2021-04-19 03:15:14.885851: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2021-04-19 03:15:14.886174: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2021-04-19 03:15:14.886454: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found

Minimal code to reproduce the issue:

import tensorflow as tf
from TicTacToeSharpEnvironmentWrapper import TicTacToeEnvironment
env = TicTacToeEnvironment()
physical_devices = tf.config.list_physical_devices('GPU')

With TicTacToeSharpEnvironmentWrapper.py

import tensorflow as tf
from tf_agents.environments import py_environment
from tf_agents.specs import BoundedArraySpec
from tf_agents.trajectories.time_step import StepType
from tf_agents.trajectories.time_step import TimeStep
import numpy as np

assembly_path1 = r"C:\DesktopGym\bin\Debug"
import sys

sys.path.append(assembly_path1)
import clr
clr.AddReference("GymSharp")
from GymSharp import TicTacToeSharpEnvironment
"""A CSharp environment for Tic-Tac-Toe game."""
class TicTacToeEnvironment(py_environment.PyEnvironment):
  """A state-settable environment for Tic-Tac-Toe game.
  """

def __init__(self):
	super(TicTacToeEnvironment, self).__init__()
	self.sharp_env = TicTacToeSharpEnvironment()

TicTacToeSharpEnvironment is a c# class libary compiled as 64bit dll

public class TicTacToeSharpEnvironment
{
	static TicTacToeSharpEnvironment()
	{
		PythonInitiliazer.InitializePython();
	}
}

And PythonInitiliazer is used to initalize Python.Net

public class PythonInitiliazer
{
	static PythonInitiliazer()
	{
		InitializePython();
	}
	static bool initialized;
	public static void InitializePython()
	{
		if (!initialized)
		{
			initPython();
			initialized = true;
		}
	}
	private static void initPython()
	{

		string pathToVirtualEnv = @"C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\";

		Environment.SetEnvironmentVariable("PATH", pathToVirtualEnv, EnvironmentVariableTarget.Process);
		Environment.SetEnvironmentVariable("PYTHONHOME", pathToVirtualEnv, EnvironmentVariableTarget.Process);
		Environment.SetEnvironmentVariable("PYTHONPATH", $"{pathToVirtualEnv}\\Lib\\site-packages;{pathToVirtualEnv}\\Lib;{pathToVirtualEnv}\\scripts", EnvironmentVariableTarget.Process);
		Runtime.PythonDLL = "C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Python37_64\\python37.dll";

		PythonEngine.PythonHome = pathToVirtualEnv;
		PythonEngine.PythonPath = Environment.GetEnvironmentVariable("PYTHONPATH", EnvironmentVariableTarget.Process);
		PythonEngine.Initialize();
	}
}

The full code works. The Python Wrapper of the C# environment passes the Tensorflow Agents unit tests for the Tic Tac Toe environment. The C# environment can be wrapped as Python or a Tensor flow environment and the various agents can train against the environment.

I don’t think this is a compatibility issue using the x64 .Net DLL because I am using 64 bit python, but I a can’t be certain.

[lostmsu]

This is not a support forum.

[alexhiggins732]

I updated the wiki Python.Net in a virtual environment so others don’t experience the same issue.

Have a nice day!