[BUG-REPORT] (Major) vaex hdf5 incompatible with number 1.22.4

Thank you for reaching out and helping us improve Vaex!

Before you submit a new Issue, please read through the documentation. Also, make sure you search through the Open and Closed Issues - your problem may already be discussed or addressed.

Description Vaex cannot export hdf5 files with numpy 1.22.4 arrays

Software information

• Vaex version (import vaex; vaex.__version__): {'vaex-core': '4.9.1', 'vaex-hdf5': '0.12.1'}
• Vaex was installed via: pip / conda-forge / from source pip
• OS: Mac/linux

Additional information The following works with numpy 1.22.3 but fails with numpy 1.22.4

import vaex

print(vaex.__version__)  # {'vaex-core': '4.9.1', 'vaex-hdf5': '0.12.1'}

df =vaex.example()
df.export("file.hdf5")

In [3]: df.export("file.hdf5")
---------------------------------------------------------------------------
BufferError                               Traceback (most recent call last)
<ipython-input-3-c4ce35d9bf95> in <module>
----> 1 df.export("file.hdf5")

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/vaex/dataframe.py in export(self, path, progress, chunk_size, parallel, fs_options, fs)
   6694             self.export_feather(path, parallel=parallel, fs_options=fs_options)
   6695         elif naked_path.endswith('.hdf5'):
-> 6696             self.export_hdf5(path, progress=progress, parallel=parallel)
   6697         elif naked_path.endswith('.fits'):
   6698             self.export_fits(path, progress=progress)

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/vaex/dataframe.py in export_hdf5(self, path, byteorder, progress, chunk_size, parallel, column_count, writer_threads, group, mode)
   6910             with Writer(path=path, group=group, mode=mode, byteorder=byteorder) as writer:
   6911                 writer.layout(self, progress=progressbar_layout)
-> 6912                 writer.write(
   6913                     self,
   6914                     chunk_size=chunk_size,

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/vaex/hdf5/writer.py in __exit__(self, *args)
     40
     41     def __exit__(self, *args):
---> 42         self.close()
     43
     44     def layout(self, df, progress=None):

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/vaex/hdf5/writer.py in close(self)
     32     def close(self):
     33         if self.mmap is not None:
---> 34             self.mmap.close()
     35             self.file.close()
     36         self.h5.close()

BufferError: cannot close exported pointers exist

[erwanp]

We also install it from Mamba (side note : it took a few days before it was available although it worked via conda yet).

It may be a different error, it looks very similar.

Fails after a

df = Vaex.open()
df.export_hdf5() 
df.close()

Exact code : https://app.travis-ci.com/github/radis/radis/jobs/572584074#L2099

Error (and package versions)

BufferError: cannot close exported pointers exist 

https://app.travis-ci.com/github/radis/radis/jobs/572584074#L2099

Note it does NOT fail during export(), but during close(). Maybe I shouldn’t close the file?

[erwanp]

@JovanVeljanoski we apparently still have the problem above; which locks us into using numpy<=1.22.3. Do you have any idea of the source ?