Wide DataFrames

Recently a few people have asked about dataframes that have many thousands of columns.

Generally the current design of dask.dataframe doesn’t work well in this case; we partition on the index, not on different columns.

We could consider partitioning on columns though, this would be a completely separate design but might be useful. I suspect that this approach would be simpler to implement for the cases where it is relevant. By assuming that any column fits into memory most algorithms become fairly simple.

[mrocklin]

A lot of people have wide dataframes, regardless of tidiness. This might not be a data engineering choice but instead perhaps a data storage choice. I can imagine storing all of my timeseries data for each stock separately and not wanting to reshuffle it all just to make my data tidy.

[saulshanabrook]

I would also like to be able to partition a dataframe by columns. In my case, I am doing natural language processing on a large number of documents. Each document has some metadata about it and also the full text (or some representation of it in tokens or parts of speech). On disk, there is one csv file for the metadata and one file for each full text.

It would be helpful if I could store all of this in one dask dataframe, that only loaded the text when that column was needed. For example, a common task is first filtering on some metadata and then getting the text for those documents. Currently, it isn’t possible to do this without reading in all of the documents for rows we are scanning.