get_dummies not executing

Dask get_dummies “runs” but it doesn’t actually execute the get_dummies task.

import pandas as pd
import dask.dataframe as dd

pandasData = pd.DataFrame({'var1': ['a', 'b', 'a'], 'var2': ['b', 'a', 'c'], 'var3': ['c', 'a', 'b']})
pd.get_dummies(pandasData)

daskData = dd.from_pandas(pandasData, npartitions=1)
daskData.head()

daskDataDummies = dd.get_dummies(daskData).compute()
daskDataDummies.head()

daskDataDummies.to_csv('daskDataDummies_out.csv', header=True, index=False)

There’s no error message it simply doesn’t transform the dataframe.

[mrocklin]

Adding to that parametrized test has been helpful so far. Next time I take a look at this I’ll probably dump in a bunch more dtypes and such.

[mrocklin]

Sampling probably isn’t sufficient. We need to know all of the values throughout the file to determine the columns.