pythonnet.load crashes if the Python interpreter is called from PATH in Ubuntu 20.04

Environment

• Pythonnet version: 3.0.0-dev (latest master on 18/2/2021)
• Python version: 3.8.5
• Operating System: Ubuntu 20.04
• .NET Runtime: .NET 5.0.2

Details

• Describe what you were trying to get done. I am trying to use the latest pythonnet in a virtual environment. If I call my simple demo script using python command in PATH, pythonnet.load() crashes. It is the same with any other PATH-based pythonX command. Using a full path works fine. See the details and the analysis below.

Preparation

# Create virtual env.
python3 -m virtualenv venv
. venv/bin/activate
# Install the latest pythonnet
git clone https://github.com/pythonnet/pythonnet.git
pip install ./pythonnet/
# Get .NET 5 runtime
wget https://download.visualstudio.microsoft.com/download/pr/824175f2-5577-46ec-a675-95d2f652575f/07dafc2398e669afa7d25f2d47398c36/dotnet-runtime-5.0.2-linux-x64.tar.gz
mkdir dotnet_5.0.2
tar xfzv dotnet-runtime-5.0.2-linux-x64.tar.gz -C dotnet_5.0.2/

Runtime config (cat runtimeconfig.json)

{
  "runtimeOptions": {
    "tfm": "net5.0",
    "framework": {
      "name": "Microsoft.NETCore.App",
      "version": "5.0.2"
    }
  }
}

Python script (cat bug.py):

import os

import clr_loader
import pythonnet

runtime_config = os.path.join(os.path.dirname(__file__), "runtimeconfig.json")
dotnet_root = os.path.join(os.path.dirname(__file__), "dotnet_5.0.2")

runtime = clr_loader.get_coreclr(runtime_config=runtime_config, dotnet_root=dotnet_root)

pythonnet.set_runtime(runtime)
pythonnet.load()

import System
from System import String

s = String("Hello World from Python.NET for .NET 5!")
System.Console.WriteLine(s)

• What commands did you run to trigger this issue? If you can provide a Minimal, Complete, and Verifiable example this will help us understand the issue.

> which python
/home/takacs/src/pythonnet/venv/bin/python

>`which python` bug.py 
Hello World from Python.NET for .NET 5!
 
> python bug.py 
Traceback (most recent call last):
  File "bug.py", line 12, in <module>
    pythonnet.load()
  File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/pythonnet/__init__.py", line 50, in load
    _FFI.dlopen(libpython, posix.RTLD_NODELETE | posix.RTLD_LOCAL)
  File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 150, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
  File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 832, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
  File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 827, in _load_backend_lib
    raise OSError(msg)
OSError: cannot load library '/home/takacs/src/pythonnet/python': /home/takacs/src/pythonnet/python: cannot open shared object file: No such file or directory.  Additionally, ctypes.util.find_library() did not manage to locate a library called '/home/takacs/src/pythonnet/python'

• If there was a crash, please include the traceback here.

Traceback (most recent call last):
  File "bug.py", line 12, in <module>
    pythonnet.load()
  File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/pythonnet/__init__.py", line 50, in load
    _FFI.dlopen(libpython, posix.RTLD_NODELETE | posix.RTLD_LOCAL)
  File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 150, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
  File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 832, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
  File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 827, in _load_backend_lib
    raise OSError(msg)
OSError: cannot load library '/home/takacs/src/pythonnet/python': /home/takacs/src/pythonnet/python: cannot open shared object file: No such file or directory.  Additionally, ctypes.util.find_library() did not manage to locate a library called 

Analysis

This issue is in the pythonnet.find_libpython module. The dlinfo.dli_fname string contains the actual command that was used to start the Python interpreter. In the case of pure python command, os.path.realpath fails the properly translate this back to the real executable.

https://github.com/pythonnet/pythonnet/blob/0f5e7814c68b846cd89019f2e23fed69eaa59eca/pythonnet/find_libpython/__init__.py#L90)

Adding a few debug lines properly demonstrate this:

def _linked_libpython_unix():
    libdl = ctypes.CDLL(ctypes.util.find_library("dl"))
    libdl.dladdr.argtypes = [ctypes.c_void_p, ctypes.POINTER(Dl_info)]
    libdl.dladdr.restype = ctypes.c_int

    dlinfo = Dl_info()
    retcode = libdl.dladdr(
        ctypes.cast(ctypes.pythonapi.Py_GetVersion, ctypes.c_void_p),
        ctypes.pointer(dlinfo))
    if retcode == 0:  # means error
        return None

    print(f"DEBUG dlinfo.dli_fname.decode(): {dlinfo.dli_fname.decode()}")
    path = os.path.realpath(dlinfo.dli_fname.decode())
    print(f"DEBUG path: {path}")
    
    if path == os.path.realpath(sys.executable):
        return None
    return path

>  ./venv/bin/python bug.py 
DEBUG dlinfo.dli_fname.decode(): ./venv/bin/python
DEBUG path: /usr/bin/python3.8
Hello World from Python.NET for .NET 5!

> ../pythonnet/venv/bin/python bug.py 
DEBUG dlinfo.dli_fname.decode(): ../pythonnet/venv/bin/python
DEBUG path: /usr/bin/python3.8
Hello World from Python.NET for .NET 5!

>python bug.py 
DEBUG dlinfo.dli_fname.decode(): python
DEBUG path: /home/takacs/src/pythonnet/python
Traceback (most recent call last):
...

The last case demonstrates that os.path.realpath just extends python with the current path which is obviously incorrect.

As a result, pythonnet.load() method’s libpython variable will point to a library which cannot be loaded (https://github.com/pythonnet/pythonnet/blob/0f5e7814c68b846cd89019f2e23fed69eaa59eca/pythonnet/__init__.py#L50). However, if this part is disabled, the application will crash later in the .NET Python.Runtime for the same reason.

[ktbarrett]

@filmor See https://github.com/ktbarrett/find_libpython/pull/32 and https://github.com/conda/conda-build/issues/2738. Py_ENABLED_SHARED isn’t a reliable flag to test, and ctypes.pythonapi can return a handle to the current Python executable. I suggest vendoring in find_libpython@master, or just add it as a dependency. I have a CI system in place now that should prevent any breaks.

[filmor]

Okay, I’m giving up on this for now, can you create a PR with your change? I’m still not happy with it, but dladdr is just (consistently) broken in this regard and there doesn’t seem to be a way to get the real argv[0] (which is what dli_fname contains at least with glibc and the BSD libcs) in a running Python process.