Can't start Kubernetes Cluster

Hi there!

Having issues starting a Kubernetes cluster using the dask-labextension and unsure if it’s a bug or some mis-configuration. The symptoms are very similiar to the Slurm issue here https://github.com/dask/dask-labextension/issues/78 . When creating a cluster using the UI it errors with

Cluster Start Error 'LocalCluster' object has no attribute 'scheduler'

I am able to create Kubernetes cluster using dask_kubernetes directly fine and scale it up and down etc, however looking to be able to do this through the UI if possible for ease of use. I have followed the documentation in terms of putting the the dask.yaml file in the appropriate directory and I know it’s being read as I have mis-configured it purposely and it error-ed out.

Any help would be really appreciated, I’ve tried to capture all of the relevant information below;

Conda Packages

dask                      2.5.0                    pypi_0    pypi
dask-core                 2.4.0                      py_0    conda-forge
dask-gateway              0.3.0                    py37_0    conda-forge
dask-kubernetes           0.9.2                      py_0    conda-forge
dask-labextension         1.0.3                    pypi_0    pypi
distributed               2.5.1                    pypi_0    pypi

Jupyter Lab Extensions

JupyterLab v1.0.4
Known labextensions:
   app dir: /opt/conda/share/jupyter/lab
        @jupyter-widgets/jupyterlab-manager v1.0.2  enabled  OK
        @jupyterlab/git v0.8.2  enabled  OK
        @jupyterlab/plotly-extension v1.0.0  enabled  OK
        dask-labextension v1.0.0  enabled  OK
        jupyter-leaflet v0.11.2  enabled  OK
        nbdime-jupyterlab v1.0.0  enabled  OK

dask.yaml

distributed:
  logging:
    bokeh: critical
  scheduler:
    work-stealing: True
    allowed-failures: 5
  admin:
    tick:
      limit: 5s

kubernetes:
  name: dask-test
  worker-template:
    spec:
      restartPolicy: Never
      containers:
      - args:
          - dask-worker
          - --nthreads
          - '1'
          - --no-bokeh
          - --memory-limit
          - "1GB"
          - --death-timeout
          - '60'
        image: daskdev/dask:latest
        name: dask-test
        resources:
          limits:
            cpu: 0.2
            memory: "1G"

labextension:
  factory:
    module: dask_kubernetes
    class: KubeCluster
    args: []
    kwargs: {}

Thanks very much.

[iwalmsley]

Just an update that using a real Kubernetes cluster (1.14.4) this does seem to be working fine.

Thanks for the great work.

[iwalmsley]

Same error as before with 0.10.0

distributed.nanny - ERROR - Timed out connecting Nanny '<Nanny: None, threads: 1>' to scheduler 'tcp://172.17.0.11:45533'
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/distributed/nanny.py", line 309, in instantiate
    timedelta(seconds=self.death_timeout), self.process.start()
tornado.util.TimeoutError: Timeout
distributed.dask_worker - INFO - End worker

Edit: To confirm - it doesn’t work either using the Labextension or just the plain code above with 0.10.0, so I guess it’ll be worth me trying in a real Kubernetes cluster.