some proxy classes dissapear

I have a reproducible test scenario. I overwrite System.setOut() and System.setErr() using two Java objects that each wrap a Python object accessing sys.stderr / sys.stdout. See https://github.com/terrier-org/pyterrier/blob/master/pyterrier/bootstrap.py#L85

I do some writing from Java to System.err, which calls back to the Python stderr stream, and works fine; The Java code then writes to System.out, which segfaults:

The relevant line in jnius_proxy.pxi is ret = py_obj.invoke(method, *py_args), where py_obj = <object><void *>jptr. I checked jptr - both have the correct value, but the cast back for the stdout doesnt work.

I have also seen

AttributeError: 'dict' object has no attribute 'invoke'
Traceback (most recent call last):
  File "jnius/jnius_proxy.pxi", line 156, in jnius.invoke0
  File "jnius/jnius_proxy.pxi", line 124, in jnius.py_invoke0

which others have reported. It appears that one of the Python objects that has been proxied is disappearing (I thought the Python object might be garbage collected, but even keeping a global reference to it doesnt help).

Do we know why the pointer can become invalid? I can try to develop a simplified test case.

Interestingly, if I keep a dict mapping the <long long> pointer back to the actual Python object, all works fine. This is illustrated in the patch below. This obviously keeps the Python object around for ever, but perhaps thats OK?

diff --git a/jnius/jnius_proxy.pxi b/jnius/jnius_proxy.pxi
index 0a1325c..8e04fdf 100644
--- a/jnius/jnius_proxy.pxi
+++ b/jnius/jnius_proxy.pxi
@@ -90,7 +90,8 @@ cdef jobject py_invoke0(JNIEnv *j_env, jobject j_this, jobject j_proxy, jobject
     ptrField = j_env[0].GetFieldID(j_env,
         j_env[0].GetObjectClass(j_env, j_this), "ptr", "J")
     jptr = j_env[0].GetLongField(j_env, j_this, ptrField)
-    py_obj = <object><void *>jptr
+    py_obj = proxy_register[jptr]
+    #py_obj = <object><void *>jptr
 
     # extract the method information
     method = Method(noinstance=True)
@@ -149,6 +151,7 @@ cdef jobject py_invoke0(JNIEnv *j_env, jobject j_this, jobject j_proxy, jobject
     except Exception:
         traceback.print_exc()
 
+cdef dict proxy_register = {}
 
 cdef jobject invoke0(JNIEnv *j_env, jobject j_this, jobject j_proxy, jobject
         j_method, jobjectArray args) with gil:
@@ -174,6 +177,10 @@ cdef create_proxy_instance(JNIEnv *j_env, py_obj, j_interfaces, javacontext):
     invoke_methods[0].fnPtr = <void *>&invoke0
     j_env[0].RegisterNatives(j_env, nih.j_cls, <JNINativeMethod *>invoke_methods, 1)
 
+    print("Registering proxy for %s with interfaces %s, pointer %d"  % (py_obj.__class__, j_interfaces, <long long><void *>py_obj))
+    proxy_register[<long long><void *>py_obj] = py_obj
+    print(str(proxy_register))
+
     # create the proxy and pass it the invocation handler
     cdef JavaClass j_obj
     if javacontext == 'app':

[cmacdonald]

Yes, that matches with what I figured out. Indeed, keeping a global ref to the Python object fixes the issue. This commit fixes it: https://github.com/terrier-org/pyterrier/commit/156f68d2498f5da51102e2c667234a0d37f79b9c

I can make a PR for documentation change, then we can close the issue. Thanks both

[cmacdonald]

GCd == garbage collected. The Python object goes out of scope to the Python interpreter, even if a reference to it still exists in Java land. See example of @tshirtman above.

See also https://github.com/kivy/pyjnius/blob/master/docs/source/api.rst#java-lambda-implementation-in-python-using-lambdas-and-function-references, particularly “You must hold a reference to the Python lambda for as long as it will be used by Java.”