Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Issues with AKS 1.22.6 using chart from main, no workers, computations hang

See original GitHub issue

What happened:

Deploying helm chart from current main and rebuilt stock dask-gateway and dask-gateway-server images on AKS 1.22.6 hangs when trying to execute any computation. I’m not sure where it’s going wrong and I don’t see anything obvious in the log files.

Edit: I managed to get debug on the scheduler and it looks like it might be a loop calling: “handler identity”. Updated those logs in this issue.

What you expected to happen: No hanging!

Minimal Complete Verifiable Example: docker build/push dask-gateway[-server] to myregistry.azurecr.io

config.yaml 
gateway:
  auth:
    simple: 
      password: test
  loglevel: DEBUG
  image:
    name: myreg.azurecr.io/jupyterhub/dask-gateway-server
    tag: latest
    pullPolicy: Always

  backend:
    # The image to use for both schedulers and workers.
    image:
      name: myreg.azurecr.io/jupyterhub/dask-gateway
      tag: latest
      pullPolicy: Always

  extraConfig:
    # Note that the key name here doesn't matter. Values in the
    # `extraConfig` map are concatenated, sorted by key name.
    clusteroptions: |
        from dask_gateway_server.options import Options, Integer, Float, String

        def option_handler(options):
            return {
                "worker_cores": options.worker_cores,
                "worker_memory": "%fG" % options.worker_memory,
                "image": options.image,
            }

        c.Backend.cluster_options = Options(
            Integer("worker_cores", 2, min=1, max=4, label="Worker Cores"),
            Float("worker_memory", 4, min=1, max=8, label="Worker Memory (GiB)"),
            String("image", default="myreg.azurecr.io/jupyterhub/dask-gateway:latest", label="Image"),
            handler=option_handler,  
        )

controller:
  loglevel: DEBUG
  image:
    name: myreg.azurecr.io/jupyterhub/dask-gateway-server
    tag: latest
    pullPolicy: Always
traefik:
  loglevel: DEBUG
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "load-balancers"
pod list

image

Sample Code. Hangs at compute() 
from dask_gateway import BasicAuth, Gateway
gateway = Gateway("http://10.15.32.135", auth=BasicAuth("mike", "test"))
gateway.list_clusters()
uster = gateway.new_cluster()
client = cluster.get_client()
import dask.array as da

a = da.random.normal(size=(1000, 1000), chunks=(500, 500))
a.mean().compute()
dask-scheduler pod logs 
distributed.preloading - INFO - Import preload module: dask_gateway.scheduler_preload
distributed.scheduler - INFO - -----------------------------------------------
distributed.preloading - INFO - Import preload module: dask_gateway.scheduler_preload
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -   Scheduler at:    tls://10.15.40.76:8786
distributed.scheduler - INFO -   dashboard at:                     :8787
distributed.preloading - INFO - Run preload setup click command: dask_gateway.scheduler_preload
distributed.comm.tcp - DEBUG - Incoming connection from 'tls://10.15.40.128:32940' to 'tls://10.15.40.76:8786'
distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2
distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms
distributed.comm.tcp - DEBUG - TLS connection with 'tls://10.15.40.128:32940': protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256
distributed.core - DEBUG - Connection from 'tls://10.15.40.128:32940' to Scheduler
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32940': {'op': 'subscribe_worker_status'}
distributed.core - DEBUG - Calling into handler subscribe_worker_status
distributed.comm.tcp - DEBUG - Incoming connection from 'tls://10.15.40.128:32942' to 'tls://10.15.40.76:8786'
distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2
distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms
distributed.comm.tcp - DEBUG - TLS connection with 'tls://10.15.40.128:32942': protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256
distributed.core - DEBUG - Connection from 'tls://10.15.40.128:32942' to Scheduler
distributed.comm.tcp - DEBUG - Incoming connection from 'tls://10.15.40.128:32944' to 'tls://10.15.40.76:8786'
distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2
distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms
distributed.comm.tcp - DEBUG - TLS connection with 'tls://10.15.40.128:32944': protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256
distributed.core - DEBUG - Connection from 'tls://10.15.40.128:32944' to Scheduler
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32942': {'op': 'register-client', 'client': 'Client-e8925bc9-bcf9-11ec-9db7-3adfd639750b', 'reply': False, 'versions': {'host': {'python': '3.8.13.final.0', 'python-bits': 64, 'OS': 'Linux', 'OS-release': '5.4.0-1073-azure', 'machine': 'x86_64', 'processor': 'x86_64', 'byteorder': 'little', 'LC_ALL': 'en_US.UTF-8', 'LANG': 'en_US.UTF-8'}, 'packages': {'python': '3.8.13.final.0', 'dask': '2022.02.0', 'distributed': '2022.02.0', 'msgpack': '1.0.3', 'cloudpickle': '2.0.0', 'tornado': '6.1', 'toolz': '0.11.2', 'numpy': '1.21.5', 'pandas': '1.3.5', 'lz4': None, 'blosc': None}}}
distributed.core - DEBUG - Calling into handler add_client
distributed.scheduler - INFO - Receive client connection: Client-e8925bc9-bcf9-11ec-9db7-3adfd639750b
distributed.core - INFO - Starting established connection
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'dump_state', 'reply': True}
distributed.core - DEBUG - Calling into handler _to_dict
distributed.comm.tcp - DEBUG - Incoming connection from 'tls://10.15.40.128:32950' to 'tls://10.15.40.76:8786'
distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2
distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms
distributed.comm.tcp - DEBUG - TLS connection with 'tls://10.15.40.128:32950': protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256
distributed.comm.tcp - DEBUG - Incoming connection from 'tls://10.15.40.128:32948' to 'tls://10.15.40.76:8786'
distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2
distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms
distributed.comm.tcp - DEBUG - TLS connection with 'tls://10.15.40.128:32948': protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256
distributed.core - DEBUG - Connection from 'tls://10.15.40.128:32950' to Scheduler
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'broadcast', 'msg': {'op': 'dump_state', 'exclude': ()}, 'on_error': 'return_pickle', 'reply': True}
distributed.core - DEBUG - Calling into handler broadcast
distributed.core - DEBUG - Connection from 'tls://10.15.40.128:32948' to Scheduler
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'versions', 'packages': (), 'reply': True}
distributed.core - DEBUG - Calling into handler versions
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'broadcast', 'msg': {'op': 'versions', 'packages': ()}, 'on_error': 'ignore', 'reply': True}
distributed.core - DEBUG - Calling into handler broadcast
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32944': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32948': {'op': 'identity', 'reply': True}
distributed.core - DEBUG - Calling into handler identity
distributed.core - DEBUG - Message from 'tls://10.15.40.128:32950': {'op': 'identity', 'reply': True}
Controller Logs 
[D 2022-04-15 20:12:07.563 KubeController] Event - ADDED cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:07.563 KubeController] Reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:07.614 KubeController] Creating new credentials for cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:07.651 KubeController] Creating scheduler pod for cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:07.667 KubeController] Event - ADDED pod dask.dask-scheduler-7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:07.676 KubeController] Event - MODIFIED pod dask.dask-scheduler-7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:07.683 KubeController] Finished reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:07.683 KubeController] Event - MODIFIED cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:07.684 KubeController] Reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:07.684 KubeController] Finished reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:07.706 KubeController] Event - MODIFIED pod dask.dask-scheduler-7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:09.479 KubeController] Event - MODIFIED pod dask.dask-scheduler-7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:09.480 KubeController] Reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:09.480 KubeController] Creating scheduler service for cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:09.506 KubeController] Event - MODIFIED cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:09.507 KubeController] Finished reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:09.507 KubeController] Reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:09.508 KubeController] Finished reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:09.535 KubeController] Event - ADDED endpoints dask.dask-7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:10.489 KubeController] Event - MODIFIED pod dask.dask-scheduler-7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:10.489 KubeController] Reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:10.490 KubeController] Finished reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:10.495 KubeController] Event - MODIFIED endpoints dask.dask-7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:10.496 KubeController] Reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:10.496 KubeController] Creating scheduler HTTP route for cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:10.505 KubeController] Creating scheduler TCP route for cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:10.539 KubeController] Event - MODIFIED cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:10.540 KubeController] Finished reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:10.540 KubeController] Reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:12:10.541 KubeController] Finished reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:15:34.005 KubeController] Too old resourceVersion in endpoints informer, relisting
[D 2022-04-15 20:15:34.011 KubeController] Relisted endpoints informer - 0 deletes, 1 updates
[I 2022-04-15 20:15:34.012 KubeController] Reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[I 2022-04-15 20:15:34.012 KubeController] Finished reconciling cluster dask.7e63d5481ec74b4d90bee1437d8829a5
api logs 
[I 2022-04-15 20:12:07.104 DaskGateway] 200 GET /api/v1/options 0.530ms
[W 2022-04-15 20:12:07.544 DaskGateway] 401 POST /api/v1/clusters/ 0.570ms
[I 2022-04-15 20:12:07.547 DaskGateway] Creating cluster dask.7e63d5481ec74b4d90bee1437d8829a5 for user mike
[I 2022-04-15 20:12:07.563 DaskGateway] 201 POST /api/v1/clusters/ 17.786ms
[D 2022-04-15 20:12:07.564 DaskGateway] Event - ADDED cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:07.564 DaskGateway] Updating dask.7e63d5481ec74b4d90bee1437d8829a5 in cache
[D 2022-04-15 20:12:07.683 DaskGateway] Event - MODIFIED cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:07.684 DaskGateway] Updating dask.7e63d5481ec74b4d90bee1437d8829a5 in cache
[D 2022-04-15 20:12:09.506 DaskGateway] Event - MODIFIED cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:09.507 DaskGateway] Updating dask.7e63d5481ec74b4d90bee1437d8829a5 in cache
[D 2022-04-15 20:12:10.540 DaskGateway] Event - MODIFIED cluster dask.7e63d5481ec74b4d90bee1437d8829a5
[D 2022-04-15 20:12:10.540 DaskGateway] Updating dask.7e63d5481ec74b4d90bee1437d8829a5 in cache
[I 2022-04-15 20:12:10.541 DaskGateway] 200 GET /api/v1/clusters/dask.7e63d5481ec74b4d90bee1437d8829a5?wait 2975.451ms
[D 2022-04-15 20:12:11.624 DaskGateway] 200 GET /api/health 0.329ms

Anything else we need to know?:

Environment:

  • Dask version:
  • Python version:
  • Operating System:
  • Install method (conda, pip, source):
Cluster Dump State: This doesn't return anything

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
TomAugspurgercommented, Apr 16, 2022

That would need to be discussed in distributed, but IMO the current behavior is appropriate. Perhaps a warning, but even then I’m not sure.

0reactions
jabberacommented, Apr 15, 2022

Might I suggest adding some sort of error when submitting a job with no workers?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshoot Azure Kubernetes Services issues
Documentation for troubleshooting common issues with Azure Kubernetes Services. ... Learn how to use AKS with these quickstarts, tutorials, and samples.
Read more >
[AKS 1.22] HttpApplicationRouting Addon is not working #2616
What happened: HttpApplicationRouting Addon doesn't work on 1.22 cluster due to the deprecation of the ingresses.extesnions API.
Read more >
Google Cloud release notes | Documentation
The following release notes cover the most recent changes over the last 60 days. For a comprehensive list of product-specific release notes, see...
Read more >
Deploy on Azure Kubernetes Service (AKS) using Helm chart
Step 4: Install YugabyteDB using Helm chart. You need to perform a number of steps to deploy YugabyteDB using Helm chart: Add the...
Read more >
Azure Kubernetes Service (AKS) Upgrades - Jonathan - Medium
kubernetes version means the worker nodes' version ... If we use CIDR to represent this number, it would be x.x.x.x/22. You could use...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

fresh azure helm deploy stuck on ImagePullBackOff, chartpress did not run?

Next page

Connecting to Dask Gateway via NGINX reverse proxy - cluster.get_client() results in TimeoutError (because gateway:// TCP traffic blocked)