Rewrite dd.read_hdf

HDF format is probably the most mature cross-platform format in common use today for pandas-like data. Dask.dataframe supports this format through dd.to_hdf and dd.read_hdf functions, which are decently full featured.

However, as various features have glommed onto read_hdf it has become fairly complex, causing significant maintenance burden and additional friction when adding new features. As we think about naturally supporting intelligent partitioning (see #1407), a new and important feature, we might also consider how read_hdf might be rewritten from scratch to improve both the existing situation and ease future development.

It would be nice to now collect both some current frustrations and some long-term desires for dealing with dask.dataframe and HDF files together so that future designs can have these in mind.

cc @nirizr @alberthdev

[ankravch]

Hi there, Is there plan for dd.read_hdf to support ‘where’, ‘columns’ keywords by exploiting what’s originally done in pd.read_hdf?

[mrocklin]

My pseudocode above fails to properly divide single large files:

If `*` is somewhere in the inputs:
    filenames, datapaths = find_all_matching_datapaths(filename, datapath)
    filenames, datapaths = sort_by_name(filenames, datapaths)
    dfs = [dd.read_hdf(fn, datapath) for fn, datapath in zip(filenames, datapaths)]
    if not mostly_sorted_divisions(dfs):
        dfs = [df.clear_divisions() for df in dfs]
    return dd.concat(dfs)  # use logic in concat to realign slightly overlapping partitions
else:
    return old_single_datapath_version_of_read_hdf(filename, datapath)

Because we still have the problem of “how do I find rows to cleanly divide up this single file”

This could be resolved if instead of these lines:

filenames, datapaths = find_all_matching_datapaths(filename, datapath)
filenames, datapaths = sort_by_name(filenames, datapaths)
dfs = [dd.read_hdf(fn, datapath) for fn, datapath in zip(filenames, datapaths)]

We had these lines

filenames, datapaths, kwargs_list = find_all_pd_read_hdf_calls(filename, datapath)
filenames, datapaths, kwargs_list = sort_by_name(filenames, datapaths, kwargs_list)
dfs = [dd.read_hdf(fn, datapath, **kwargs) for fn, datapath, kwargs in zip(filenames, datapaths, kwargs_list)]

Where now kwargs_list might optionally include start/stop entries for the pd.read_hdf call. That is we make a new dask.dataframe for every partition, even for partitions within a single file/datapath.