closed connection?

I am running Dask Cluster on k8s, and just bumped to daskdev/dask:2.8.0 for scheduler, custom image with the below specs for workers.

- dask=2.8.1
- distributed=2.8.1 
- dask-glm=0.2.0
- dask-kubernetes=0.9.2
- dask-ml=1.1.1
- dask-xgboost=0.1.9

Both are running python 3.7. My client has the same settings and python, and code below does not show any issues

client = Client('tcp://dask-adaptive-scheduler.staging2.mydomain.cloud:8786')
client.get_versions(check=True);

But when I run the dask_ml.model_selection.RandomizedSearchCV, my dashboard shows there is no workers. When I check pods, there is a worker called dask-adaptive-dask-workerttlcv, which throws the error below in stdout:

tornado.application - ERROR - Exception in callback <function Worker._register_with_scheduler.<locals>.<lambda> at 0x7f8bb81779d8>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
    return self.callback()
  File "/opt/conda/lib/python3.7/site-packages/distributed/worker.py", line 855, in <lambda>
    lambda: self.batched_stream.send({"op": "keep-alive"}),
  File "/opt/conda/lib/python3.7/site-packages/distributed/batched.py", line 117, in send
    raise CommClosedError

where should I dig? Previous version, dask=2.4.0, worked fine with the same k8s settings

[mrocklin]

that seems like a different error regardless though. It seems like the original issue here was a non-issue. I’m going to close this. If you think that you’ve come across something new, the let’s raise a new issue.

[mrocklin]

Then I must be incorrect.

On Wed, Nov 27, 2019 at 11:51 AM Philipp Kats [e-mail] wrote:

Thanks!

Just to make sure Which master are you referring to? Couse I only use official daskdev/dask image, which I suspecd wasn’t updated to 2.8.1 😬

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/3272?email_source=notifications&email_token=AACKZTGOUI7GNUXPGRQY3LTQV3FTVA5CNFSM4JR66E6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFKSVKA#issuecomment-559229608, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTCU2L5A7EW6V74DMNDQV3FTVANCNFSM4JR66E6A .