Local port forwarding failing

Describe the issue:

I’m consistently getting “kubectl port forward failed” from port_forward_service using the standard KubeCluster class to create an ad hoc cluster. When this happens the port forward is successfully setup and I can connect via the port that is opened, but the is_comm_open check returns false. sock.connect_ex((ip, port)) is always returning code 22 which is essentially “invalid input”. If I change the is_comm_open check to:

if await is_comm_open("", local_port, retries=100):  # pass an empty string instead of localhost

it began to temporarily work, but after about 30 minutes that also began to fail.

Minimal Complete Verifiable Example:

I’m seeing this on a very basic setup (environment posted below):

pod_spec = make_pod_spec(image='ghcr.io/dask/dask:latest',
                         memory_limit='4G', memory_request='4G',
                         cpu_limit=1, cpu_request=1)

n_workers = 4
with KubeCluster(
    pod_template=pod_spec,
    name="test",
    n_workers=3
) as cluster:

Environment:

• Dask version: 2022.9.1
• Dask-kubernetes version: 2022.7.0
• Python version: 3.8
• Operating System: MacOS 12.6
• Install method (conda, pip, source): conda

[semnooij]

That seems to resolve it for me as well. I’d suggest the following instead of the explicit closure.

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    result = sock.connect_ex((ip, port))

[jacobtomlinson]

Thanks for looking at this @baswelsh I’d love to see a PR with this. Totally agree with @semnooij that a context manager is a nice way to handle this.