Python path does not seem to include DLLs directory under Windows

Environment

• Pythonnet version: 3.0.1
• Python version: 3.11
• Operating System: Windows
• .NET Runtime: 7.0.4

Details

• Describe what you were trying to get done.

  Load the _socket module.

  I’m willing to believe that this is something specifically wrong in my environment, though I have no idea what it could be. On MacOS, PythonEngine.PythonPath includes the “lib-dynload” subdirectory which includes _socket.dylib. But on Windows, it does not include the “DLLs” subdirectory containing _socket.pyd. By contrast, if I create a simple C program which just calls Py_Initialize() and then gets Py_GetPath(), it does include “DLLs”.

  When I use PROCMON to see what the process is doing, it is reading a registry key which has the right path in it.

• What commands did you run to trigger this issue? If you can provide a Minimal, Complete, and Verifiable example this will help us understand the issue.

    PythonEngine.Initialize();
    using (Py.GIL())
    {
        dynamic ss = Py.Import("_socket");
    }

• If there was a crash, please include the traceback here.

N/A

[CurtHagenlocher]

Okay, if nothing else I’m leaving this here as a message for anyone else who runs into this problem.

I went ahead and debugged through the Python startup code (which looks very different than the last time I did this in the Python 1.4 or 1.5 era 😉 and came to the conclusion that the issue is in Python itself. The problem seems to be that Windows is a bit of an outlier when it comes to loading platform-specific native modules. Non-Windows platforms seem to load these from a location relative to the rest of the Python library, while on Windows they’re loaded from a location relative to the currently-running executable. This means that for embedding scenarios (like Python.NET), you can’t just reference an existing Python installation (e.g. via PYTHONNET_PYDLL), you have to either copy files to the right places or you have to manually amend sys.path to find the actual “platlibdir” on Windows.

Without knowing why Python chose this behavior – and the startup code mentions all of “QUIRK” and “compatibility” and “security” in this area – my gut reaction is that the behavior is wrong. But it’s also something that realistically couldn’t be changed until Python 3.13 (and that’s if someone is actually interested in changing it and has the time to do so).

For context, I ran into this problem because the Apache Arrow project is using Python.NET to test interoperability between the C# Arrow implementation and the C Arrow API, and the existing tests simply don’t run under Windows. This is unfortunate. Obviously, it can be worked around in that codebase but given that it’s Python.NET which is embedding Python it feels “more right” to me to have Python.NET mitigate between the behavior of Python and people’s expectations.

That said, the backwards-compatibility concerns are no less important here, and I can appreciate not wanting to change the code for an edge case which my internet search skills suggest there really isn’t anyone hitting. This is a really neat project and I appreciate that it exists. I worked on IronPython (and tangentially on C# dynamic) at Microsoft nearly 15 years ago, and this is exactly the kind of capability we were hoping to enable with C# dynamic. Thanks @lostmsu !

[CurtHagenlocher]

Do you have a link to the current test suite?

https://github.com/apache/arrow/blob/main/csharp/test/Apache.Arrow.Tests/CDataInterfacePythonTests.cs

It’s just a small part of the overall tests, but it’s a probably a good approach to keep in mind. (Also I’m really only just starting with contributions there so I’m in no position to speak for what others might think is a good idea.)