Add bar chart for worker occupancy

We currently have bar charts for worker memory and the number of tasks processing. A similar chart to number of tasks processing would be worker occupancy, which records the sum of the amount of compute time of all tasks currently residing on the worker. This would be exactly like the number of tasks processing chart, but use ws.occupancy rather than len(ws.processing)

I would have used this today when trying to diagnose work stealing issues.

The relevant code for the number of processing tasks is here. It’s made slightly complex because the code for this chart is currently merged with the memory chart. It might make sense to break these apart.

https://github.com/dask/distributed/blob/9d4f0bf2fc804f955a869febd3b51423c4382908/distributed/dashboard/components/scheduler.py#L1017-L1139

[jrbourbeau]

Is there anything we can add here to make this more useful, or is a standalone page in the vein of /individual-* good here?

That’s a good question. For now I think an /individual-occupancy plot would be a good start. I think @ncclementi is currently working on adding this

[mrocklin]

I think that this chart should probably look like the charts on the status page, with number of processing tasks and the amount of memory used. Both are horizontally aligned bar-charts that update sub-second-ish. We probably want all three of these plots to be somewhat consistent.

On Tue, Jun 22, 2021 at 4:50 PM James Bourbeau @.***> wrote:

Since occupancy is in units of time

https://github.com/dask/distributed/blob/6d15b013ebe59f07e6f563cf73a4e8a7268d260c/distributed/scheduler.py#L442-L446

I would expect the units of the bars (i.e. the x-axis) to be time and for that to be periodically updated. Is that what you mean by varying on time, or do you mean something else?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/4870#issuecomment-866360707, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTD2GXKVMKJM3G2LYKDTUEATNANCNFSM456YRPNQ .