Problem accessing the dashboard when the cluster frontend has an internal and an external IP

Funnily enough I helped solve the problem in https://github.com/dask/dask-jobqueue/issues/71 but I now realise that the dashboard is not accessible in our SGE cluster and the ssh tunneling work-around I proposed does not cut it.

Even more fun, when I use dask-drmaa, I can access the dashboard fine (a ssh tunnel is needed of course).

Here is my current understanding:

• I run the scheduler on the cluster frontend called sequoia
• the cluster frontend has a external IP, which is used for example when I ssh from my local machine to the frontend:

# from my local machine
$ ping sequoia
PING sequoia.paris.inria.fr (128.93.90.8) 56(84) bytes of data.

• the cluster frontend has an internal IP that is used when I am on the cluster:

# from sequoia i.e. the cluster frontend
$ ping sequoia
PING sequoia.cm.cluster (10.141.255.254) 56(84) bytes of data.

• the scheduler is using the internal IP address which means that even with ssh tunneling, the dashboard is not going to be reachable. Here is an excerpt of client._repr_html_() for completeness:

Scheduler: </b>tcp://10.141.255.254:57591\n
Dashboard: </b><a href=\'http://10.141.255.254:45961/status\' target=\'_blank\'>http://10.141.255.254:45961/status</a>

In some way it feels like some kind of variation of https://github.com/dask/dask-drmaa/issues/75:

• in my case (using dask-jobqueue), the internal IP is used by the scheduler so the dashboard is not accessible from the outside.
• in their case (using dask-drmaa), the external IP was used by the scheduler but this was a problem because the workers can only access the internal IP of the frontend. cc @MaxNoe in case I misunderstood something. In particular I would be very interested to see client._repr_html_() on their setup.

Inputs more than welcome:

• is my cluster setup quite unusual? A bit of a simplification, but changing our cluster setup is not a very likely solution.
• do we need to have the scheduler use the internal IP and the dashboard the external IP (if that is even is)?
• other things I may have missed ?

I’d be more than happy to provide more info if that can help!

[mrocklin]

Note that all of Dask’s pages use relative links and should be fine. It’s just the bokeh index that requires to be told the prefix. In practice just not showing users the index page works pretty well

On Thu, Jun 28, 2018 at 3:58 AM, jakirkham [e-mail] wrote:

Yep, it advertises itself as a JupyterHub project, but was able to get it to work with the Jupyter Notebook by merely installing it. Should add the conda-forge package runs the jupyter-serverextension part for you as part of the install. So if using conda, just installing it seems to work (for me at least 😃.

Assuming default Jupyter Notebook port and Dask Dashboard port, the URL looks something like this, http://localhost:8888/proxy/8787/, or more generally, http://localhost:<notebook_port>/proxy/<dask_dashboard_port>/. This drops you on the application menu of Dask Dashboard. The status page lives at http://localhost:<notebook_port>/proxy/<dask_dashboard_ port>/status.

There is a small issue about links on the Dask Dashboard main page being wrong. However this comes up in other cases as well. So the problem is not unique to nbserverproxy and may require a little work on Distributed side of things. ( dask/distributed#2073 https://github.com/dask/distributed/issues/2073 ) Should add the problem does not extend to the application header bar shown when a single application is running.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dask/dask-jobqueue/issues/80#issuecomment-400947329, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszPhVTBsrlurIFRnLzbjFsiG9ttoJks5uBIy7gaJpZM4U51ty .

[guillaumeeb]

There is some misunderstanding here I believe. Command you should try:

ssh -fN lesteve@sequoia -L 45961:10.141.255.254:45961