How to set index and devisions lazily using DataFrame.read_csv?

The DataFrame.read_csv method currently does not accept an index_col argument. Instead, the error message suggests to use dd.read_csv(...).set_index('my-index') instead. However, this is an expensive calculation, which should not be necessary if the index is already sorted and the divisions are known.

Is there a way to explicitly set index and divisions using the read_csv method, that does not require to load in data?

[abast]

Perfect! However, I do not only want to set the divisions, but the index as well. Overall, I am struggling for quite a while now with the very basic task to save a dask dataframe to disk in parallel and reload it lazily:

• setting an index using read_csv is expensive (even with sorted = True)
• hdf apparently does not support very wide dataframes
• the to_hdf method requires some special attention to be able to save in parallel
• fastparquet will be a supreme solution but currently has the disclaimer of beeing not battle tested.

I therefore figured, others might be in the same situation that they would like to provide as much metadata as possible to the read_csv function and wanted to suggest to extend the API of this method in a way, that known divisions along a specified index column can be provided.

[mrocklin]

If your column is sorted then you can use df.set_index(column, sorted=True). This will still involve a full pass through the data to find locations, but will not require an on-disk shuffle.

You might also consider moving to a format like Parquet, which allows for index information to be written as metadata. This is in master in dd.io.parquet but not yet released. http://fastparquet.readthedocs.io/en/latest/details.html#connection-to-dask