Objects in Map<String,Object> and other complex containers sometimes disappear

I am using pyjnius to pass nested Python dictionaries and lists to Java Sometimes the content of Map<String,Object> and ArrayList<Object> disappears.

It usually goes okay with simple one-dimensional lists and dictionaries filled with strings and integers. But when structures get more complex sometimes data goes missing. It doesn’t always go wrong, one day a certain Python object maps fine to Java, the other day the resulting Java-objects misses items or is completely empty.

I’m using the following versions: Python 2.7.6 Cython 0.23.4 java version “1.7.0_95” OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.1) OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) Tried jnius 1.0.2 and 1.1-dev

import jnius
from collections import Iterable, Mapping
import gc

# Disable the Python garbage-collector, just to be sure
gc.disable()

# Java DataTypes
jMap       = jnius.autoclass('java.util.HashMap')
jArrayList = jnius.autoclass('java.util.ArrayList')
jInt       = jnius.autoclass('java.lang.Integer')
jLong      = jnius.autoclass('java.lang.Long')
jFloat     = jnius.autoclass('java.lang.Float')
jDouble    = jnius.autoclass('java.lang.Double')
jString    = jnius.autoclass('java.lang.String')

class JavaNumber(object):
    '''
    Convert int/float to their corresponding Java-types based on size
    '''
    def __call__(self, obj):
        if isinstance(obj, int):
            if obj <= jInt.MAX_VALUE:
                return jInt(obj)
            else:
                return jLong(obj)
        elif isinstance(obj, float):
            if obj < jFloat.MAX_VALUE:
                return jFloat(obj)
            else:
                return jDouble(obj)

# Map between Python types and Java
javaTypeMap = { int:   JavaNumber(),
                str:   jString,
                float: JavaNumber() }


def mapObject(data):
    '''
    Recursively convert Python object to Java Map<String, Object>
    :param data:
    '''
    try:
        if type(data) in javaTypeMap:
            # We know of a way to convert type
            return javaTypeMap[type(data)](data)
        elif isinstance(data, jnius.MetaJavaClass):
            # It's already a Java thingy, or None
            return data
        elif isinstance(data, Mapping):
            # Object is dict-like
            map = jMap()
            for key, value in data.iteritems():
                map.put(str(key), mapObject(value))
            return map
        elif isinstance(data, Iterable):
            # Object is list-like
            array = jArrayList()
            for item in data:
                i = mapObject(item)
                array.add(i)
            return array
        else:
            # Convert it to a String
            return jString(str(data))
    except:
        print 'Failed to map Python-object to Java!'
        print str(data)
        raise



goes_okay = {'list': {3: 4, 5: 6}, 
              4      : 5 }

misses_dict_item = {'list': [1, 2, 7], 
                      'int': 3, 
                      4: 5, 
                      'dict': {2: 3} }

empty = {'access_log': [{'stored_proc': 'getsomething'},
                        {'uses': [{'usedin': 'some->bread->crumb'},
                                {'usedin': 'something else here'},
                                {'stored_proc': 'anothersp'}]},
                        {'uses': [{'usedin': 'blahblah'}]}],
        'reporting': [{'stored_proc': 'reportingsp'},
                    {'uses': [{'usedin': 'breadcrumb'}]}]}

print mapObject(goes_okay).toString()
print mapObject(misses_dict_item).toString()
print mapObject(empty).toString()

<bountysource-plugin> --- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/32616881-objects-in-map-string-object-and-other-complex-containers-sometimes-disappear?utm_campaign=plugin&utm_content=tracker%2F77133&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F77133&utm_medium=issues&utm_source=github). </bountysource-plugin>

[AKuederle]

here is the reference to the other issue: https://github.com/kivy/pyjnius/issues/345

[AKuederle]

That’s awesome! I will have a look. From my naive knowledge about Python, it appears to an issue with the garbage collection. Basically, the second you leave the current scope the object gets deleted. It feels like this might be connected to the bug with the refcounter increase reported in another bug. I am currently at mobile but I will link the issue tomorrow.

On Wed, Nov 28, 2018, 21:03 Curtis Rueden <[e-mail] wrote:

@AKuederle https://github.com/AKuederle I implemented conversion methods similar to those discussed here, and published them on PyPI as part of the scyjava https://pypi.org/project/scyjava/ Python module. Could you give it a try and see if any of your conversions suffer from this issue? I wrote quite a few unit tests and was unable to find any failing cases. I avoided the issue by splitting the function calls across multiple lines of code, as shown by the above minimal example; additionally, I use LinkedHashMap rather than HashMap, which does not seem to suffer from the problem regardless.

I’d love to hear any ideas from others on why the following works:

HashMap = autoclass(‘java.util.HashMap’) hm_good = HashMap() k = new_key() v = new_val() hm_good.put(k, v)print('good: ’ + hm_good.toString())

But this does not:

hm_bad = HashMap() hm_bad.put(new_key(), new_val())print('bad: ’ + hm_bad.toString())

I am not a Python expert, but I find it hard to believe that a bug like this could really be on the Pyjnius side.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kivy/pyjnius/issues/217#issuecomment-442585508, or mute the thread https://github.com/notifications/unsubscribe-auth/AKRwuyFVEMSsNRfmQFdHRX7F24x_MaaJks5uzuwNgaJpZM4IACro .