Passing file objects into `h5py.File` is tricky

• Operating System Windows 10
• Python version 3.6
• Built with pyenv
• h5py version (2.10 & 3.0.0rc1)
• HDF5 version 1.12.0

Summary of the h5py configuration
---------------------------------

h5py    3.0.0rc1
HDF5    1.12.0
Python  3.6.10 (default, Mar 20 2020, 00:41:24) 
[GCC 7.5.0]
sys.platform    linux
sys.maxsize     9223372036854775807
numpy   1.17.4
cython (built with) 0.29.21
numpy (built against) 1.12.0
HDF5 (built against) 1.12.0

I thought you could create HDF5 files from file objects but I get “I/O operation on closed file” errors. What’s going wrong here?

>>> with open("test.hdf5", "w") as f:
...  h = h5py.File(f, "w")
...  h.create_group("ehe")
... 
Traceback (most recent call last):
  File "h5py/_objects.pyx", line 199, in h5py._objects.ObjectID.__dealloc__
  File "h5py/h5fd.pyx", line 169, in h5py.h5fd.H5FD_fileobj_write
ValueError: I/O operation on closed file.
Exception ignored in: 'h5py._objects.ObjectID.__dealloc__'
Traceback (most recent call last):
  File "h5py/_objects.pyx", line 199, in h5py._objects.ObjectID.__dealloc__
  File "h5py/h5fd.pyx", line 169, in h5py.h5fd.H5FD_fileobj_write
ValueError: I/O operation on closed file.
<HDF5 group "/ehe" (0 members)>

[Helveg]

Ok, turns out there are no real actionable issues here. There’s 3 things going on:

1. Me not opening file handles in binary mode
2. The context managers (with statements) closing the underlying file handle
3. h5py v3.0.0 changing the behavior of missing file modes in h5py.File to an error that I had not seen before since I just switched to 3.0.0

---

1. This is mentioned in the docs

The file-like object must be open for binary I/O

1. Also mentioned in the docs

Accessing the File instance after the underlying file object has been closed will result in undefined behaviour.

1. Very unfortunate as this broke the documentation example (#1702) strengthening my belief there was a large collection of errors surrounding file objects

---

In summary, be sure to:

• Open your file handle with read/write permissions in binary mode: f = open(filename, "wb+")
• Explicitly specify a file mode argument to the h5py.File constructor: h = h5py.File(f, "w")
• Close your h5py.File handle BEFORE you close your file handle

The correct way to pass a file object to h5py.File:

f = open("test.h5", "wb+")
h = h5py.File(f, "w")
# ...
h.close()
f.close()

With with:

with open("test.h5", "wb+") as f:
  with h5py.File(f, "w") as h:
    # ...

I think there are some improvements that can be made:

• Instead of allowing undefined behavior, throw an exception when a closed file handle is encountered
• Instead of the long and cryptic error message, use the default Python “TypeError: File() is missing required argument mode” (https://github.com/h5py/h5py/issues/1703)
• When closing the File object and the underlying handle is already closed, there’s a segfault

[takluyver]

Using file-like objects with h5py was new in version 2.9. The docs do say that. 😉

https://docs.h5py.org/en/stable/high/file.html#python-file-like-objects