from_hdf5 function that uses filenames rather than files

The current approach to load data from HDF5 is to load in an h5py or netcdf4 Dataset object and pass that to from_array. This is efficient, but also includes the file pointer within the dask graph. This fails if you need to serialize the graph, such as is necessary in distributed computing.

It might be wise to instead create an explicit da.from_hdf5 function that only stored the filename, datapath, and slice information within the graph.

It would also be nice if this API was exposed up to xarray (cc @shoyer).

cc @rabernat @pwolfram

[DaanVanVugt]

@mrocklin I took your hdf5lazy approach a bit further by wrapping all h5py objects to support __{set,get}state__ and to remember how we opened the file (especially mode is important). An LRU cache is also implemented. This allows me to replace import h5py with import h5pickle as h5py and have working serialization in the dask distributed scheduler.

See the code at https://github.com/Exteris/h5pickle

This works very well for me in dask. Would you mind taking a look at the code? Perhaps this could be useful in distributed.

ps. Thanks for making dask so awesome

[jcrist]

A few ideas:

• Expanding your idea above, signature could be da.from_hdf5(filepath, datapath). Slice is composed later, as it feels more natural to index (also, : only works in getitem). Example:

z1 = da.from_hdf5('foo.h5', '/x/y/z1')[:, :100]
z2 = da.from_hdf5('foo.h5', '/x/y/z2')[:, :100]

• Another approach would be to make a “lazy” version of the h5py File objects that don’t open the file pointer until explicitly asked to (or closes and reopens the file pointer when serialized). Example:

x = da.from_hdf5('foo.h5')
z1 = x['/x/y/z1'][:, :100]
z2 = x['/x/y/z2'][:, :100]