ctypes.pythonapi issues; getting AttributeError: undefined symbol

Using the new python3 toolchain, I am trying to use PyCryptodome (3.7+) as a dependency.

As in https://github.com/kivy/python-for-android/pull/1685, PyCryptodome >= 3.6.4 runs into compilation issues; I am trying to fix these too, though this is not that relevant here.

PyCryptodome >=3.6.0 crashes at runtime (since commit https://github.com/Legrandin/pycryptodome/commit/f5aa2c1618e97b6d773172fdd07794a0a6f05905). So actually the currently pinned version in the recipe does not work: https://github.com/kivy/python-for-android/blob/80e4f059c1ee0da48a7c85167087dfe5928ac395/pythonforandroid/recipes/pycryptodome/__init__.py#L5

The issue at runtime is with ctypes.

Say I have a main script that just does the following (https://github.com/Legrandin/pycryptodome/blob/95ccce7ae82d3a36f1a8652dd2c645222d0128dd/lib/Crypto/Util/_raw_api.py#L200):

import ctypes
ctypes.pythonapi.PyObject_GetBuffer

This works with cpython on my laptop, but with the p4a-compiled python on Android it fails:

06-14 19:06:27.053 15246 15274 I python  : Android kivy bootstrap done. __name__ is __main__
06-14 19:06:27.053 15246 15274 I python  : AND: Ran string
06-14 19:06:27.053 15246 15274 I python  : Run user program, change dir and execute entrypoint
06-14 19:06:27.092 15246 15274 I python  : Traceback (most recent call last):
06-14 19:06:27.092 15246 15274 I python  :   File "/home/user/wspace/electrum/.buildozer/android/app/main.py", line 84, in <module>
06-14 19:06:27.093 15246 15274 I python  :   File "/home/user/wspace/electrum/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ctypes/__init__.py", line 369, in __getattr__
06-14 19:06:27.093 15246 15274 I python  :   File "/home/user/wspace/electrum/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ctypes/__init__.py", line 374, in __getitem__
06-14 19:06:27.094 15246 15274 I python  : AttributeError: undefined symbol: PyObject_GetBuffer

I have also tried to access some other attributes of ctypes.pythonapi, such as Py_IncRef, which raises the same exception.

Any idea what’s going on here? Why is ctypes.pythonapi non-functional?

[AndreMiras]

And I could confirm the issue with ctypes.

08-24 00:52:08.844 16982 17327 I python  : Run user program, change dir and execute entrypoint
08-24 00:52:08.866 16982 17327 I python  : ctypes.__version__: 1.1.0
08-24 00:52:08.909 16982 17327 I python  : Traceback (most recent call last):
08-24 00:52:08.909 16982 17327 I python  :   File "/home/andre/workspace/python-for-android/issues/1866/.buildozer/android/app/main.py", line 10, in <module>
08-24 00:52:08.911 16982 17327 I python  :   File "/home/andre/workspace/python-for-android/issues/1866/.buildozer/android/app/main.py", line 6, in main
08-24 00:52:08.912 16982 17327 I python  :   File "/home/andre/workspace/python-for-android/issues/1866/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ctypes/__init__.py", line 369, in __getattr__
08-24 00:52:08.913 16982 17327 I python  :   File "/home/andre/workspace/python-for-android/issues/1866/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ctypes/__init__.py", line 374, in __getitem__
08-24 00:52:08.914 16982 17327 I python  : AttributeError: undefined symbol: PyObject_GetBuffer
08-24 00:52:08.914 16982 17327 I python  : Python for android ended.

Code was:

import ctypes


def main():
    print(f'ctypes.__version__: {ctypes.__version__}')
    print(f'ctypes.pythonapi.PyObject_GetBuffer: {ctypes.pythonapi.PyObject_GetBuffer}')


if __name__ == '__main__':
    main()

Edit: This is in the C-API https://docs.python.org/3/c-api/buffer.html#c.PyObject_GetBuffer https://github.com/python/cpython/blob/3.7/Objects/abstract.c#L353 hence should be part of libpython3.7.so. And confirmed on Ubuntu 18.04:

readelf -s /usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7.so | grep PyObject_GetBuffer
  1857: 000000000027b960    68 FUNC    GLOBAL DEFAULT   12 PyObject_GetBuffer

And in the p4a build.

readelf -s .buildozer/android/platform/build/dists/myapp/build/intermediates/jniLibs/debug/armeabi-v7a/libpython3.7m.so | grep PyObject_GetBuffer
   315: 0003b8e4    84 FUNC    GLOBAL DEFAULT   11 PyObject_GetBuffer

So it’s as if we were not linking to it.

[pakal]

Initially, in pycryptodome, ctypes is used as fallback to cffi, because of the “ImportError: CFFI with optimize=2 fails due to pycparser bug.”

Is there any way to make cffi work in python-for-android, so that pycryptodome can work despite ctypes issues? I’ve seen in the current master some cmd line params like “no_compile_pyo” and “optimize_python”, but I don’t really get how/when they are used in the compilation toolchain (the pycryptodome files I find in the build directory are “pyc” and not “pyo”, but they seem to be stripped from their docstrings anyway).

On the other hand, the fallback from cffi to ctypes, above, looks at sys.flags.optimize only, not the real state of some files… is there any way to make p4a run the python interpreter in non optimized mode, to see if it solves problems?

EDIT: I forced both flags in p4a to skip bytecode optimization, but files compiled into “_python_bundle/” are still without docstring, and anyway the python interpreter seems to be forced to use PYTHONOPTIMIZED=2 in multiple parts of the toolchain… so cffi seems unusable.