Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
`CommClosedErrors` resulting from `Client.shutdown()`
See original GitHub issueDescribe the issue:
I am experiencing CommClosedError errors during the shutdown of a manually constructed Dask cluster using the Dask CLI. This was first noted in dask/dask-mpi#94.
Minimal Complete Verifiable Example:
You need a minimum of 3 open terminal sessions (e.g., bash) to reproduce the error.
In Terminal 1:
$ dask-scheduler
Note the ADDRESS in the Scheduler at: ADDRESS:8786 log message.
In Terminal 2:
$ dask-worker ADDRESS:8786
where ADDRESS is the scheduler IP address (without the port number).
In Terminal 3:
$ python
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:36:39) [GCC 10.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from distributed import Client
>>> client = Client('ADDRESS:8786')
>>> client.shutdown()
where ADDRESS is the scheduler IP address noted above.
Results
In Terminal 1, the scheduler shuts down appropriately without errors.
In Terminal 2, the worker shuts down, but not without error. The logs of the worker after client.shutdown() is called are:
2022-10-25 15:54:14,502 - distributed.worker - INFO - Stopping worker at ADDRESS:40141
2022-10-25 15:54:14,503 - distributed.worker - INFO - Connection to scheduler broken. Closing without reporting. ID: Worker-ce265b58-eecf-4335-bd1e-d0fc3b07e93d Address ADDRESS:40141 Status: Status.closing
2022-10-25 15:54:14,503 - distributed.batched - INFO - Batched Comm Closed <TCP (closed) Worker->Scheduler local=ADDRESS:33682 remote=ADDRESS:8786>
Traceback (most recent call last):
File ".../lib/python3.10/site-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File ".../lib/python3.10/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File ".../lib/python3.10/site-packages/distributed/comm/tcp.py", line 269, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
2022-10-25 15:54:14,507 - distributed.nanny - INFO - Worker closed
2022-10-25 15:54:14,508 - distributed.nanny - ERROR - Worker process died unexpectedly
2022-10-25 15:54:14,699 - distributed.nanny - INFO - Closing Nanny at 'ADDRESS:46611'.
2022-10-25 15:54:14,699 - distributed.dask_worker - INFO - End worker
In Terminal 3, where the client is running, a CommClosedError appears every time the client heartbeat is called (about once every 5 seconds):
2022-10-25 15:54:15,825 - tornado.application - ERROR - Exception in callback <bound method Client._heartbeat of <Client: 'ADDRESS:8786' processes=1 threads=8, memory=7.63 GiB>>
Traceback (most recent call last):
File ".../lib/python3.10/site-packages/tornado/ioloop.py", line 905, in _run
return self.callback()
File ".../lib/python3.10/site-packages/distributed/client.py", line 1390, in _heartbeat
self.scheduler_comm.send({"op": "heartbeat-client"})
File ".../lib/python3.10/site-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Client->Scheduler local=ADDRESS:33802 remote=ADDRESS:8786> already closed.
And this does not stop repeating until the Python process is exited (e.g., exit()).
Anything else we need to know?:
Interestingly, you can avoid the error that appears in Terminal 2 in the worker logs if you call client.retire_workers() before calling client.shutdown(), but the CommClosedError errors in the Client application (Terminal 3) are still present.
NOTE: The CommClosedError messages in the worker logs appears to have been introduced in version 2022.4.2. These errors do not appear in version 2022.4.1 or 2022.4.0.
Environment:
- Dask version: 2022.10.0 --> 2022.4.2 / Client heartbeat
CommClosedErrorsappear much further back - Python version: 3.10.X, 3.9.X, 3.8.X
- Operating System: Linux, Windows
- Install method (conda, pip, source): conda (from conda-forge)
Issue Analytics
- State:
- Created a year ago
- Comments:13 (2 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Here’s
dask_mpi.initialize(exit=False):Log
I think all those errors are spurious, though. If I don’t let the scheduler and worker fall through to start a new client, then I get no errors. Meaning
main()looks like this:Then I get this log with no errors:
Log
Yes, I can use this workaround. Thanks!