Crash with pypy

I tried running things with pypy and got crashes. They are weird and this is the minimal I got:

import fiona
import json

with fiona.open("ne_10m_admin_0_countries.shp") as source:
    countries = list(source)

with open("randomfilenameiuiubiqubeiquequze", "w") as sink:
    for country in countries:
        sink.write(json.dumps(country)+"\n")

The shapefile is from the latest http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_countries.zip

Running this with pypy3 I get three different tracebacks, seemingly on random.

RPython traceback: File “rpython_jit_metainterp_1.c”, line 41233, in assembler_call_helper_65 File “rpython_jit_metainterp_8.c”, line 21619, in AbstractResumeGuardDescr_handle_fail File “rpython_jit_metainterp_8.c”, line 30933, in AbstractResumeGuardDescr__trace_and_compile_from File “rpython_jit_metainterp_8.c”, line 38152, in MetaInterp_handle_guard_failure File “rpython_jit_metainterp_8.c”, line 41192, in MetaInterp__handle_guard_failure File “rpython_jit_metainterp_1.c”, line 5322, in MetaInterp__interpret File “rpython_jit_metainterp_2.c”, line 22084, in MIFrame_run_one_step File “rpython_jit_metainterp_4.c”, line 13161, in handler_ref_return_1 File “rpython_jit_metainterp_4.c”, line 61659, in MetaInterp_finishframe *** Invalid usage of a dying CPython object ***

cpyext, the emulation layer, detected that while it is calling an object’s tp_dealloc, the C code calls back a function that tries to recreate the PyPy version of the object. Usually it means that tp_dealloc calls some general PyXxx() API. It is a dangerous and potentially buggy thing to do: even in CPython the PyXxx() function could, in theory, cause a reference to the object to be taken and stored somewhere, for an amount of time exceeding tp_dealloc itself. Afterwards, the object will be freed, making that reference point to garbage.

PyPy could contain some workaround to still work if you are lucky, but it is not done so far; better fix the bug in the CPython extension. Aborted (core dumped)

or

RPython traceback: File “rpython_jit_metainterp_2.c”, line 50548, in handle_jitexception_68 *** Invalid usage of a dying CPython object *** (…)

or

RPython traceback: File “pypy_module_pypyjit.c”, line 183, in portal_12 File “pypy_interpreter_2.c”, line 40731, in handle_bytecode__AccessDirect_None File “pypy_interpreter_2.c”, line 51912, in dispatch_bytecode__AccessDirect_None *** Invalid usage of a dying CPython object *** (…)

Fiona Fiona-1.7.11.post1-cp36-cp36m-manylinux1_x86_64.whl pypy3 says: Python 3.5.3 (3f6eaa010fce78cc7973bdc1dfdb95970f08fed2, Feb 12 2018, 22:20:44) [PyPy 5.10.1 with GCC 7.3.0] on linux gdal 2.2.3 Archlinux

[rbuffat]

@sgillies The counter argument would be, that somebody has some rather large computations, that he wants to speed up with pypy, and then export the results with fiona. But I think this will be a rather limited use case.

[micahcochran]

If you are only using a shapefile, you might try pyshp, which is python-only code. Currently, it isn’t tests for pypy, but it might not be too hard to get it to working.