Hang in unit test test_async.py::test_versions

What happened:

Attempted to run the dask-kubernetes unit test suite. The test suite appears to hang in the first test case: dask_kubernetes/tests/test_async.py::test_versions

What you expected to happen:

I expected to see all 67 test cases run w/o issue.

Minimal Complete Verifiable Example:

I was able to create the hang condition with this command from the project’s root directory:

pytest -v dask_kubernetes/tests

Anything else we need to know?:

I captured the following log output.

pytest innovcation

$ pytest -v  dask_kubernetes/tests
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /Users/jim/.conda/envs/dask-kubernetes/bin/python
cachedir: .pytest_cache
rootdir: /Users/jim/Desktop/dask/dask-kubernetes, configfile: setup.cfg
plugins: asyncio-0.16.0, xdist-2.5.0, forked-1.4.0, pycharm-0.7.0, kind-21.1.3
collected 67 items

dask_kubernetes/tests/test_async.py::test_versions

The pytest output stops at the above line.

Display status of the kind cluster with kubectl get all -A command

$ kubectl get all -A
NAMESPACE            NAME                                                    READY   STATUS             RESTARTS   AGE
default              pod/dask-jim-0bdc4023-bmqgvf                            0/1     Pending            0          10m
kube-system          pod/coredns-74ff55c5b-pvsv9                             0/1     Pending            0          11m
kube-system          pod/coredns-74ff55c5b-wjv4p                             0/1     Pending            0          11m
kube-system          pod/etcd-pytest-kind-control-plane                      1/1     Running            0          11m
kube-system          pod/kindnet-kmkr8                                       1/1     Running            3          11m
kube-system          pod/kube-apiserver-pytest-kind-control-plane            1/1     Running            0          11m
kube-system          pod/kube-controller-manager-pytest-kind-control-plane   1/1     Running            0          11m
kube-system          pod/kube-proxy-m8rp8                                    0/1     CrashLoopBackOff   7          11m
kube-system          pod/kube-scheduler-pytest-kind-control-plane            1/1     Running            0          11m
local-path-storage   pod/local-path-provisioner-78776bfc44-2l8mv             0/1     Pending            0          11m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  11m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   11m

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kindnet      1         1         1       1            1           <none>                   11m
kube-system   daemonset.apps/kube-proxy   1         1         0       1            0           kubernetes.io/os=linux   11m

NAMESPACE            NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system          deployment.apps/coredns                  0/2     2            0           11m
local-path-storage   deployment.apps/local-path-provisioner   0/1     1            0           11m

NAMESPACE            NAME                                                DESIRED   CURRENT   READY   AGE
kube-system          replicaset.apps/coredns-74ff55c5b                   2         2         0       11m
local-path-storage   replicaset.apps/local-path-provisioner-78776bfc44   1         1         0       11m

log file from the crashed pod/kube-proxy-m8rp8

$ kubectl logs pod/kube-proxy-m8rp8 -n kube-system
I0104 01:15:18.746378       1 node.go:172] Successfully retrieved node IP: 172.30.0.2
I0104 01:15:18.746449       1 server_others.go:142] kube-proxy node IP is an IPv4 address (172.30.0.2), assume IPv4 operation
I0104 01:15:18.774456       1 server_others.go:185] Using iptables Proxier.
I0104 01:15:18.775208       1 server.go:650] Version: v1.20.2
I0104 01:15:18.776129       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 163840
F0104 01:15:18.776174       1 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied

Searching on the error message led to this issue: https://github.com/kubernetes-sigs/kind/issues/2240

Environment:

• Dask version: From what I can tell, the unit test installs from the main branch.
• Python version: Python 3.9.7
• Operating System: Unit test is run on MacOS 11.6.1
• Install method (conda, pip, source): N/A
• Other software
  • Docker Desktop 4.2.0 (Docker Engine 20.10.10)
  • kubectl client 1.21.7
  • Helm 3.7.1

[jimthompson5802]

@jacobtomlinson Thank you very much for you help. I am now able to run the unit test.

It looks like my conda environment was broken. After deleting and recreating the conda environment and installing the requirements.txt and requirements-test.txt dependencies, the pytest tests now run.

Here is the pytest summary line

FAILED dask_kubernetes/tests/test_helm.py::test_discovery - AssertionError: assert 'helmcluster' in {'proxycluster': {'discover': <function discover at 0x7fb529a...
=========================================== 1 failed, 55 passed, 6 skipped, 5 xfailed, 61 warnings in 368.41s (0:06:08) ============================================

Since I can run the unit tests, I’m closing this issue. Thank you again for your help.

[jimthompson5802]

Thank you for the insight. I may have found root cause. It appears my conda enviornment may be broken. When I look in the conda env library, I see this

Let me kill the conda environment and recreate. I’ll get back to you on this.

I really appreciate your patience and guidance in tracking this down.