latest dask `map_partitions` doesn't pass list as expected.

Reproducible example

I used the HIGGS dataset from UCI machine learning reprository for demonstraction purose. But any other dataset should suffice.

The failure happens only on dask 2021.05.0, older versions work fine. Also using tuple instead of list works fine.

from distributed import LocalCluster, Client, wait
from dask import dataframe as dd


def map_partition_fn(block, meta_list):
    # Works with  dask-2.22.0, assert error with 2021.05.0
    #
    # distributed.worker - WARNING - Compute Failed
    # Function:  execute_task
    # args:      ((subgraph_callable, [(<function read_block_from_file at 0x7f84259231f0>, <OpenFile 'higgs/HIGGS.csv'>, 640000000, 64000000, b'\n'), None, False]))
    # kwargs:    {}
    # Exception: AssertionError(<class 'str'>)
    assert isinstance(meta_list, list), type(meta_list)
    return block


def main(client):
    df: dd.DataFrame = dd.read_csv("higgs/HIGGS.csv")
    meta = [i for i in range(len(df.columns))]

    mapped = df.map_partitions(map_partition_fn, meta_list=meta)
    mapped = client.persist(mapped)
    wait(mapped)


if __name__ == "__main__":
    with LocalCluster() as cluster:
        with Client(cluster) as client:
            main(client)

• Dask version: 2021.05.0
• Python version: 3.8.5
• Operating System: Ubuntu 20.04.2
• Install method (conda, pip, source): pip

cc @jakirkham

[rjzamora]

Thanks for raising this @trivialfis ! As James mentioned, the task fusion default was recently changed. I’ll look into this asap.

[jrbourbeau]

Thanks @trivialfis, I’m able to reproduce using minor.csv. I think this is related to Dask recently disabling low-level task fusion by default for DataFrames (xref https://github.com/dask/dask/pull/7620). If I turn task fusion back on with dask.config.set({"optimization.fuse.active": True}) then meta_list is a list as expected and the code snippet runs successfully

cc’ing @rjzamora in case you have any thoughts on this