question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Support for Dask Gateway clusters from config

See original GitHub issue

Right now, IIUC, to create a cluster using the button the config takes a python class, args, and kwargs to create the cluster. This isn’t flexible enough for dask-gateway, which requires creating an intermediate Gateway object.

Two options

  1. Expand the logic of the lab extension’s cluster creation to take a snippet of code to run.
  2. Update dask-gateway to have a “simple” way of creating a cluster that just uses the defaults (cc @jcrist).

https://github.com/dask/dask-gateway/issues/55 is related, but more focused on expanding dask-labextension to take advantage of dask-gateway

Issue Analytics

  • State:open
  • Created 3 years ago
  • Reactions:1
  • Comments:8 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
jcristcommented, Jun 29, 2020

dask-gateway doesn’t require creating an intermediate Gateway object - you can already call dask_gateway.GatewayCluster directly (we might want to update our docs to better show this). I’ve verified things work with dask-labextension with default parameters just like any other cluster.

0reactions
thomafredcommented, Aug 4, 2021

I see that @consideRatio found a fix using the GatewayCluster-class (https://github.com/dask/dask-labextension/issues/203). Confirmed to work with the daskhub helm-chart:

jupyterhub:
  hub:
    extraConfig:
      10-patch-dask-labextension-config: |-
        c.KubeSpawner.environment.setdefault("DASK_LABEXTENSION__FACTORY__MODULE", "dask_gateway")
        c.KubeSpawner.environment.setdefault("DASK_LABEXTENSION__FACTORY__CLASS", "GatewayCluster")
        c.KubeSpawner.environment.setdefault("DASK_LABEXTENSION__FACTORY__ARGS", "[]")
        c.KubeSpawner.environment.setdefault("DASK_LABEXTENSION__FACTORY__KWARGS", "{}")

However, the DASK DASHBOARD URL is not set correctly as mentioned by Erik in the issue above.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Exposing Cluster Options - Dask Gateway
By default cluster configuration (e.g. worker memory, docker image, etc…) ... This supports dynamic configuration options that depend on the requesting user.
Read more >
Dask-gateway: Cluster Options on K8s Setup empty
dask -gateway version is 0.9.0. EDIT: I should add that everything else works like a charm. EDIT2: I get a weird 401 on...
Read more >
Using Dask Gateway — Pangeo Gallery documentation
We've done most of the configuration for you. All that remains for you is to either create a new cluster or connect to...
Read more >
Deploy Dask Gateway with JupyterHub on Kubernetes
Preparation · Launch dask gateway · Modify the JupyterHub configuration · Create a dask cluster.
Read more >
Dask Gateway, set worker resources - Stack Overflow
How can this be done, either when the cluster is created using the config.yaml of the helm chart (ideally, a field in the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found