Pythonnet 2.5.2 in a Virtual Environment, DLL dependencies paths ignored

Environment

• Pythonnet version: 2.5.2
• Python version: 3.9.1 (32 bits)
• Operating System: Windows 10 64 bits
• .NET Runtime: >4

Details

- Describe what you were trying to get done.

I have inherited a testing setup that I have updated to the latest version of Python and Pythonnet, I am trying to import a custom DLL (32 bits, NET Runtime >4) in a Python script.

This was working with Python 3.6.5 32 bits and Pythonnet 2.3.0, however I have updated many other dependencies in the process so it might be unrelated.

- What commands did you run to trigger this issue?

I build the virtual environment as described below:

python -m venv env
.\env\Scripts\activate
python -m pip install --upgrade pip setuptools wheel --no-cache-dir
python -m pip install -r .\requirements.txt --no-cache-dir

Then I run the script from within the virtual environment, which looks for the DLLs that are added via clr.addReference, it finds these DLLs but not the related dependencies. These DLLs are stored in env/Scripts.

However, CLR seems to ignore the paths in sys.path and only looks in the Python installation directory and the location of the assembly file, thus it fails to load dependencies. If I store the DLLs in the Python installation directory it works, but it won’t work if they are stored in env/Scripts (where python.exe for the Virtual Environment exists).

- If there was a crash, please include the traceback here.

E   ModuleNotFoundError: No module named 'XXXModule'

Also, the logs from assembly binding log viewer:

Here, it finds the XXXModule.

*** Assembly Binder Log Entry  (08/02/2021 @ 16:41:01) ***

The operation was successful.
Bind result: hr = 0x0. The operation completed successfully.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable  C:\Program Files (x86)\Python39-32\python.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: Where-ref bind. Location = C:\Projects\xxx\xxx\ci\automation\python\deployment\XXXModule\XXXModule.dll
LOG: Appbase = file:///C:/Program Files (x86)/Python39-32/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = python.exe
Calling assembly : (Unknown).
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: No application configuration file found.
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Attempting download of new URL file:///C:/Projects/xxx/xxx/ci/automation/python/deployment/XXXModule/XXXModule.dll.
LOG: Assembly download was successful. Attempting setup of file: C:\Projects\xxx\xx\ci\automation\python\deployment\XXXModule\XXXModule.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: XXXModule, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
LOG: Re-apply policy for where-ref bind.
LOG: Where-ref bind Codebase does not match what is found in default context. Keep the result in LoadFrom context.
LOG: Binding succeeds. Returns assembly from C:\Projects\xxx\xxx\ci\automation\python\deployment\XXXModule\XXXModule.dll.
LOG: Assembly is loaded in LoadFrom load context.

Here it fails to find the related dependency for XXXModule:

*** Assembly Binder Log Entry  (08/02/2021 @ 16:41:01) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable  C:\Program Files (x86)\Python39-32\python.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: DisplayName = XXModuleDependency, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
 (Fully-specified)
LOG: Appbase = file:///C:/Program Files (x86)/Python39-32/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = python.exe
Calling assembly : SdpsModule, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: No application configuration file found.
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Python39-32/XXModuleDependency.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Python39-32/XXModuleDependency/XXModuleDependency.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Python39-32/XXModuleDependency.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Python39-32/XXModuleDependency/XXModuleDependency.EXE.
LOG: Attempting download of new URL file:///C:/Projects/xxx/xxx/ci/automation/python/deployment/SdpsPSModule/XXModuleDependency.DLL.
LOG: Attempting download of new URL file:///C:/Projects/xxx/xxx/ci/automation/python/deployment/SdpsPSModule/XXModuleDependency/XXModuleDependency.DLL.
LOG: Attempting download of new URL file:///C:/Projects/xxx/xxx/ci/automation/python/deployment/SdpsPSModule/XXModuleDependency.EXE.
LOG: Attempting download of new URL file:///C:/Projects/xxx/xxx/ci/automation/python/deployment/SdpsPSModule/XXModuleDependency/XXModuleDependency.EXE.
LOG: All probing URLs attempted and failed.

[dmgstz]

I managed to identify the root cause of this issue, and it comes down a bug in venv from Python 3.7 onwards, pending fix in Python 3.10 apparently, that is affecting multiple third-party modules… basically it creates a subprocess whose parent is the base python interpreter rather than the python executable in the virtual environment, more details here: https://bugs.python.org/issue38905.

Thus this is what causes the CLR to see the main Python interpreter as the running executable, instead of the python file in the virtual environment, using virtualenv instead of venv solved the issue for me.

Thought it would be interesting for you to know, as it kinds of affects the behaviour of how Pythonnet will work in virtual environments.

[dmgstz]

@filmor

Yes, tried that, but no success.

I managed to track down the issue to this, if I use Python 3.6.5 and Pythonnet 2.3.0 I get the following report:

The operation was successful.
Bind result: hr = 0x0. The operation completed successfully.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable  C:\Projects\xxx\xxx\ci\automation\python\env\Scripts\python.exe

If I run under Python 3.9.1 and Pythonnet 2.5.2, I get the following report:

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable  C:\Program Files (x86)\Python39-32\python.exe

So it comes down to this:

1. If I use Python 3.6.5 and Pythonnet 2.3.0 the running executable is C:\Projects\xxx\xxx\ci\automation\python\env\Scripts\python.exe, the one in the virtual environment, everything works fine.
2. If I use Python 3.9.1 and Pythonnet 2.5.2 the running executable is the main python executable, not the one in the virtual environment, the DLLs are located in the virtual environment thus it fails to find them.

I do not intend to use this as a support forum or annoy you, but rather as to report something odd I noticed with my setup, does this look like something that can be linked to Pythonnet? I upgraded from Python 3.6.5 and Pythonnet 2.3.0 to Python 3.9.1 and Pythonnet 2.5.2 and I am experiencing this issue with no changes in the code, it was working perfectly fine with the previous versions, that is why I reported it here with as much info I could collect and after trying many of the things in the troubleshooting guide.