Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Dashboard cannot proxy Worker Dashboards in distributed cluster
See original GitHub issueWhat happened:
Set up a distributed Dask cluster (1 scheduler and 4 workers) each running on different hosts and accessed the dashboard:
Then tried to access the Worker Dashboards via the Info tab and received the following error:
What you expected to happen:
Expected to be able to access the worker dashboards successfully
Anything else we need to know?:
On initial attempt I got a different error advising that I needed to install the additional jupyter-sever-proxy package to enable worker dashboard proxying. However once installed you get the above error because the jupyter-server-proxy package enforces a whitelist that only has localhost and 127.0.0.1 in it by default.
There does not appear to be any way for users to customise this whitelist so it is hard to see how this feature is ever expected to work except for a fully local cluster i.e. all workers on same physical machine as the scheduler.
Ideally the Dask scheduler dashboard should automatically update the proxy whitelist whenever a new worker is registered.
Environment:
- Dask version: 2.17.0
- Python version: 3.8
- Operating System: OS X
- Install method: conda
Issue Analytics
- State:
- Created 3 years ago
- Comments:11 (8 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Just curious, what is the current state of this issue? I’m using the
daskdev/dask:2021.6.2image to launch a scheduler/worker cluster inside Kubernetes and this appears to still be an issue. To be clear, I’m installing the latest version ofjupyter-server-proxyvia theEXTRA CONDA PACKAGESenvironment variable. The package installation succeeds, the cluster starts up and functions correctly, but proxying to the worker dashboards still results in a 403 error on account of whitelisting being required.Is there any configuration we can provide to the scheduler and/or workers to workaround this issue until it is resolved?
I assumed it was something to do with the server config on my end, but seeing this issue I have hope that maybe I could also be able to access worker dashboards!
To be clear, when I say “fully local” I mean a LocalCluster on a remote server that I use via jupyterlah lab in my browser; I realise my previous phrasing didn’t indicate that