BUG: compute=False not working correctly in new ORC writer

Using the new ORC writer from https://github.com/dask/dask/pull/7756 with compute=False, and then afterwards computing the result, only the first partition gets written:

import pandas as pd
import dask.dataframe as dd
import pathlib

df = pd.DataFrame(np.random.randn(100, 4), columns=['a', 'b', 'c', 'd'])
ddf = dd.from_pandas(df, npartitions=4)

In [5]: ddf.to_orc("test_orc_dataset")
Out[5]: (None, None, None, None)

In [9]: list(pathlib.Path("test_orc_dataset/").glob("*"))
Out[9]: 
[PosixPath('test_orc_dataset/part.0.orc'),
 PosixPath('test_orc_dataset/part.3.orc'),
 PosixPath('test_orc_dataset/part.2.orc'),
 PosixPath('test_orc_dataset/part.1.orc')]

In [11]: dataset = ddf.to_orc("test_orc_dataset_delayed", compute=False)

In [12]: dataset.compute()

In [13]: list(pathlib.Path("test_orc_dataset_delayed/").glob("*"))
Out[13]: [PosixPath('test_orc_dataset_delayed/part.0.orc')]

With the Parquet writer, it’s working correctly.

cc @rjzamora

[jorisvandenbossche]

Just tried to reproduce this and turns out I was on pyarrow 3 which failed at ddf.to_orc:

Yes, ORC write support was only added more recently, so maybe dask could check for the version and give a nicer error message

[jorisvandenbossche]

Yes - Sorry for all the churn here. I have a rough version of what I want to do in #8004 -

No need to be sorry! I am just genuinely interested to see how it could be organized / if I want to adapt my implementation in https://github.com/geopandas/dask-geopandas/pull/91 as well. Will try to take a look at #8004 one of the coming days.