Keep original filenames in dask.dataframe.read_csv

For the data I am reading, the path (directory name) is an important trait, and this would be useful to access (possibly as an additional column, path_as_column = True) or at the very least in the collection of delayed objects

import dask.dataframe as dd
dd.read_csv('s3://bucket_name/*/*/*.csv', collection=True)

The collection version of the command comes closer

all_dfs = dd.read_csv('s3://bucket_name/*/*/*.csv', collection=True)
print('key:', all_dfs[0].key)
print('value:', all_dfs[0].compute())

but returns an internal code as the key and doesn’t seem to have the path (s3://bucket_name/actual_folder/subfolder/fancyfile.csv) anywhere

key: pandas_read_text-35c2999796309c2c92e6438c0ebcbba4
value:    Unnamed: 0   T   N   M   count
0           0  T1  N0  M0  0.4454
1           5  T2  N0  M0  0.4076
2           6  T2  N0  M1  0.0666
3           1  T1  N0  M1  0.0612
4          10  T3  N0  M0  0.0054

[jsignell]

I have been thinking more about the path_parser idea. Pandas already accepts a converters kwargs. So we could just do a minor tweak and check that converters dict for the path_col and if we find it do the transformation before returning the df. I think the value is in only needing to do the operation once for each path vs. lots of times. This is what it would look like:

[mrocklin]

I’m inclined to skip the path_parser option suggested by @jlstevens . There is, I think, a clear path for users to handle this orthogonally on their own with the map or apply methods if they so choose. I don’t think that we need to make this any easier for them.

df = dd.read_csv(..., include_filenames=True)
df['filename'] = df['filename'].map(parse)

I think that this approach is simpler in the long run.