distributed.comm.tcp - WARNING - Closing dangling stream in <TCP> in compute() method

Hi folks, I’m working with a 64GB dataset with 625882182 lines in a SLURM cluster (I’m using one machine with 128GB of RAM as 40 workers, also the interconnection between the workers and my workstations is Infiniband). So, I need to make the median and the quantile of one column, so I’m trying to gather this column to my workstation; however, this issue occurred during the perform : distributed.comm.tcp - WARNING - Closing dangling stream in <TCP>

These are my codes: njobs = df[df.phases != '-1'].njobs

njobs is:

npartitions=1043
   int64
     ...
   ...  
     ...
     ...
Name: njobs, dtype: int64
Dask Name: getitem, 5215 tasks

So when I ran: njobs = njobs.compute()

This warning occurred: distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://XXX.XXX.XXX:55166 remote=tcp://XXX.XXX.XXX:8786>

The data was gathered correctly, but I wonder to resolve this warning.

Thanks!

[PabloPavan]

@SultanOrazbayev I didn’t, sometimes this issue occurs, sometimes don’t. But for me, the results seem correct at all. I tested using the InfiniBand and the Ethernet, and it happened in both, so I think this issue is not the interconnection.

[mrocklin]

Closing due to lack of information.