Serialization with `"msgpack"` doesn't preserve `list`s

Currently if a list is serialized (and is small enough), it will be handled correctly.

In [1]: from distributed.protocol import serialize, deserialize                 

In [2]: t = (0, 1, 2)                                                           

In [3]: deserialize(*serialize(t, serializers=["msgpack"]))                     
Out[3]: (0, 1, 2)

In [4]: l = [0, 1, 2]                                                           

In [5]: deserialize(*serialize(l, serializers=["msgpack"]))                     
Out[5]: [0, 1, 2]

However for larger lists, this breaks down and a tuple is returned instead.

In [1]: from distributed.protocol import serialize, deserialize                 

In [2]: t = (0, 1, 2, 3, 4, 5, 6)                                               

In [3]: deserialize(*serialize(t, serializers=["msgpack"]))                     
Out[3]: (0, 1, 2, 3, 4, 5, 6)

In [4]: l = [0, 1, 2, 3, 4, 5, 6]                                               

In [5]: deserialize(*serialize(l, serializers=["msgpack"]))                     
Out[5]: (0, 1, 2, 3, 4, 5, 6)

[jakirkham]

It’s worth noting msgpack itself by default will always return a list.

In [1]: import msgpack                                                          

In [2]: t = (0, 1, 2)                                                           

In [3]: msgpack.loads(msgpack.dumps(t))                                         
Out[3]: [0, 1, 2]

In [4]: l = [0, 1, 2]                                                           

In [5]: msgpack.loads(msgpack.dumps(l))                                         
Out[5]: [0, 1, 2]

That said, we actually force it to return a tuple, which presents its own set of issues.

In [6]: msgpack.loads(msgpack.dumps(l), use_list=False)                         
Out[6]: (0, 1, 2)

This choice goes back to PR ( https://github.com/dask/distributed/pull/2000 ). Though it seems we knew this could present a problem ( https://github.com/dask/distributed/pull/2000#issuecomment-396327604 ). Not sure if the reasoning behind that change still holds today or IOW what our constraints are now.

Edit: Also related is this upstream discussion ( https://github.com/msgpack/msgpack-python/issues/98 ).

[jakirkham]

Yeah that makes sense. Agree it is error-prone currently and it would be good to get away from that.

Maybe we can come up with something using ExtType?

import msgpack


def default(obj):
    if isinstance(obj, tuple):
        return msgpack.ExtType(10, msgpack.packb(list(obj)))
    else:
        raise TypeError("Unknown type: %s" % repr(type(obj)))

def ext_hook(code, data):
    if code == 10:
        return tuple(msgpack.unpackb(data))
    else:
        return msgpack.ExtType(code, data)


data = (2, 3)
packed = msgpack.packb(data, default=default, strict_types=True, use_bin_type=True)
unpacked = msgpack.unpackb(packed, ext_hook=ext_hook)
print((data, unpacked))


data = [2, 3]
packed = msgpack.packb(data, default=default, strict_types=True, use_bin_type=True)
unpacked = msgpack.unpackb(packed, ext_hook=ext_hook)
print((data, unpacked))