`KeyError` in `ProfileTimePlot.update`

Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]: 2022-08-11 09:49:57,015 - bokeh.core.property.validation - ERROR - 'start'
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]: Traceback (most recent call last):
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/utils.py", line 805, in wrapper
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     return func(*args, **kwargs)
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/dashboard/components/shared.py", line 295, in update
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     ts = metadata["keys"][self.key]
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]: KeyError: 'start'
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]: 2022-08-11 09:49:57,015 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f455b4cd0c0>>, <Task finished name='Task-5714' coro=<ServerSession.with_document_locked() done, defined at /opt/conda/envs/coiled/lib/python3.10/site-packages/bokeh/server/session.py:78> exception=KeyError('start')>)
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]: Traceback (most recent call last):
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/ioloop.py", line 741, in _run_callback
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     ret = callback()
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     future.result()
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/bokeh/server/session.py", line 95, in _needs_document_lock_wrapper
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     result = func(self, *args, **kwargs)
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/bokeh/server/session.py", line 229, in with_document_locked
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     return func(*args, **kwargs)
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/bokeh/document/callbacks.py", line 450, in wrapper
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     return invoke_with_curdoc(doc, invoke)
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/bokeh/document/callbacks.py", line 408, in invoke_with_curdoc
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     return f()
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/bokeh/document/callbacks.py", line 449, in invoke
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     return f(*args, **kwargs)
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/bokeh/document/callbacks.py", line 179, in remove_then_invoke
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     return callback()
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/dashboard/components/shared.py", line 316, in <lambda>
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     self.doc().add_next_tick_callback(lambda: self.update(prof, metadata))
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/bokeh/core/property/validation.py", line 95, in func
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     return input_function(*args, **kwargs)
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/utils.py", line 805, in wrapper
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     return func(*args, **kwargs)
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/dashboard/components/shared.py", line 295, in update
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]:     ts = metadata["keys"][self.key]
Aug 11 09:49:57 ip-10-0-12-62 cloud-init[1268]: KeyError: 'start'

(https://cloud.coiled.io/dask-engineering/clusters/48635/details)

[jrbourbeau]

Thanks @bilzard, that’s helpful. I’m able with to reproduce with the current main branch of distributed if I just create a LocalCluster, then navigate to the dashboard, and click on the empty task stream plot. @hendrikmakait do you have bandwidth to look into this one?

[mrocklin]

The task stream plot has an empty glyph on it named start when first starting. This was due to a historical bug in Bokeh.

When you click on a rectangle in the task stream plot it brings up the profile plot keyed to all profile history of tasks with that name. When you’ve first started up the scheduler without any tasks run the profile will be empty.

Probably we shouldn’t err here, but should present an empty profile plot. In practice though, this isn’t that big of an issue. It’s a bit awkward that this situation exists, but it’s entirely artificial.

On Wed, Dec 21, 2022 at 8:40 PM James Bourbeau @.***> wrote:

Thanks @bilzard https://github.com/bilzard, that’s helpful. I’m able with to reproduce with the current main branch of distributed if I just create a LocalCluster, then navigate to the dashboard, and click on the empty task stream plot. @hendrikmakait https://github.com/hendrikmakait do you have bandwidth to look into this one?

— Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/6876#issuecomment-1362322577, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTCXH7CYJQEU4PA7W7TWOO5STANCNFSM56IKYIIA . You are receiving this because you are subscribed to this thread.Message ID: @.***>