error in disrtribued fifo scheduler

If I run the distributed fifo scheduler across multiple instances I get the following error

autogluon.protocol.core - CRITICAL - Failed to deserialize 
Traceback (most recent call last): 
File "/usr/local/lib/python3.6/site-packages/distributed/protocol/core.py", line 106, in loads header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
autogluon.utils_comm - INFO - Got an unexpected error while collecting from workers: tuple is not allowed for map key

It seem to be related to this issue Is there a reason why we fix distributed==2.6.0 ?

[aaronkl]

ok it’s seems the problem went away with latest changes in the master branch

[aaronkl]

the following code snippet should reproduce the error:

import autogluon as ag
import numpy as np

dist_ip_addrs = # add list of worker IP addresses here

@ag.args(p=ag.Real(0, 1),
         epochs=10,)
def train_fn(args, reporter, **kwargs):

   for e in range(args.epochs):
        epoch = e + 1
        r = np.random.binomial(1, p=args.p)

        reporter(epoch=epoch, result=r)

scheduler = ag.scheduler.FIFOScheduler(train_fn,
                                       dist_ip_addrs=dist_ip_addrs,
                                       resource={'num_cpus': 2, 'num_gpus': 0},
                                       num_trials=100,
                                       reward_attr='result',
                                       time_attr='epoch')
scheduler.run()
scheduler.join_jobs()

the versions of the relevant dependencies are:

asn1crypto 0.24.0 attrs 19.3.0 awscli 1.18.13 boto3 1.9.176 botocore 1.12.176 catboost 0.22 ConfigSpace 0.4.10 Cython 0.29.13 dask 2.6.0 decorator 4.4.2 distributed 2.6.0 docutils 0.14 gluoncv 0.6.0 gluonnlp 0.8.1 graphviz 0.8.4 mxnet 1.6.0 mxnet-mkl 1.4.1 numpy 1.18.1 onnx 1.4.1 terminado 0.8.3

let me know if you require further information