visititems behavior changed

I’m throwing an exception from a visititems callback function, in this case to implement a timeout. This is mostly useful for hdf5 files with many items, where listing them potentially takes a long time. This worked fine with previous h5py versions, with 3.0.0 instead of the thrown Exception, it causes a SystemError:

something <HDF5 dataset "something": shape (128, 128), type "<f8">
Traceback (most recent call last):
  File "h5py/h5o.pyx", line 302, in h5py.h5o.cb_obj_simple
  File "/home/alex/.virtualenvs/hdf5test/lib/python3.8/site-packages/h5py/_hl/group.py", line 601, in proxy
    return func(name, self[name])
  File "h5test.py", line 7, in throwing
    raise Exception("yeah")
Exception: yeah

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "h5test.py", line 11, in <module>
    f.visititems(throwing)
  File "/home/alex/.virtualenvs/hdf5test/lib/python3.8/site-packages/h5py/_hl/group.py", line 602, in visititems
    return h5o.visit(self.id, proxy)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
SystemError: <built-in function visit> returned a result with an error set

This can be reproduced using the following:

import h5py
import numpy as np

if __name__ == "__main__":
    def throwing(name, obj):
        print(name, obj)
        raise Exception("yeah")

    with h5py.File("/tmp/test.h5", "w") as f:
        f['something'] = np.ones((128, 128))
        f.visititems(throwing)

With h5py 2.10.0, the Exception can be properly caught:

Traceback (most recent call last):
  File "h5test.py", line 11, in <module>
    f.visititems(throwing)
  File "/home/alex/.virtualenvs/hdf5test/lib/python3.8/site-packages/h5py/_hl/group.py", line 565, in visititems
    return h5o.visit(self.id, proxy)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 355, in h5py.h5o.visit
  File "h5py/defs.pyx", line 1641, in h5py.defs.H5Ovisit_by_name
  File "h5py/h5o.pyx", line 302, in h5py.h5o.cb_obj_simple
  File "/home/alex/.virtualenvs/hdf5test/lib/python3.8/site-packages/h5py/_hl/group.py", line 564, in proxy
    return func(name, self[name])
  File "h5test.py", line 7, in throwing
    raise Exception("yeah")
Exception: yeah

Is it possible to revert back to the previous behavior of bubbling up the original exception, or as an alternative, have native support for a timeout in visititems? Thanks!

• Operating System: Debian unstable
• Where Python was acquired: system Python

Summary of the h5py configuration
---------------------------------

h5py    3.0.0
HDF5    1.12.0
Python  3.8.5 (default, Aug  2 2020, 15:09:07) 
[GCC 10.2.0]
sys.platform    linux
sys.maxsize     9223372036854775807
numpy   1.19.4
cython (built with) 0.29.21
numpy (built against) 1.17.5
HDF5 (built against) 1.12.0

[uellue]

1881ef40bffa480e45c5579b2af656baf2a43375 is the first bad commit commit 1881ef40bffa480e45c5579b2af656baf2a43375 Author: Thomas Kluyver [e-mail] Date: Wed Dec 4 11:24:24 2019 +0000

Give Cython specific error return values to check for

:100644 100644 70351c2e9a038be04333b049e44f09cb50e4c403 708aca1c4dc687692d28757a486d8632b99f0bbd M api_gen.py :040000 040000 283885f49aeea561d6020e1d2749f3a14b7e308f 1013a1594c755dd5e5947c69ac6448da60bc9be7 M h5py

[takluyver]

Aha, that makes some sense. The visit code in the repository didn’t change, but the auto-generated bindings did. Specifically, the Cython function changed from except * to except <herr_t>-1, meaning Cython only checks for an exception if it returns -1.

I think there’s a mistake in our callback wrapper. The docs for H5Ovisit1 say that the callback return values mean:

• Zero causes the visit iterator to continue, returning zero when all group members have been processed.
• A positive value causes the visit iterator to immediately return that positive value, indicating short-circuit success.
• A negative value causes the visit iterator to immediately return that value, indicating failure.

But we return 2 on failure, instead of a negative value:

https://github.com/h5py/h5py/blob/eec7bd0ad4f7a5cbf61f5b2d4171a8e769dc5746/h5py/h5o.pyx#L277

https://github.com/h5py/h5py/blob/eec7bd0ad4f7a5cbf61f5b2d4171a8e769dc5746/h5py/h5o.pyx#L293

You’ve done the hard work of finding where the error was introduced. Do you want to do the pull request to fix it? The actual change should be straightforward - if what I said above is right - but it could do with a test and a release note.