Image pulls initiated by k3s are subject to a 2 minute timeout

Actual Behavior

Image pulls initiated by K8S result in ImagePullBackoff if the download does not complete within 2 minutes. The image pull is retried, but the pod will stay in this status forever if retries last longer than 2 minutes.

Steps to Reproduce

This test pod uses a large image:

apiVersion: v1
kind: Pod
metadata:
  name: splunktest
spec:
  containers:
  - name: splunktest
    image: splunk/splunk
    env:
    - name: SPLUNK_START_ARGS
      value: --accept-license
    - name: SPLUNK_PASSWORD
      value: password

1. Save the above yaml to a file such as test.yaml
2. Run kubectl apply -f test.yaml
3. If the connection is slow enough, the pod will enter ImagePullBackoff state after 2 minutes
4. Notice that docker pull splunk/splunk will succeed, even if it takes longer than 2 minutes

Result

When you’re developing locally and from your home office, your internet may not be fast enough to download multi-gigabytes docker images in under 2 minutes. And for smaller images, you’re subject to a shaky internet connection (family might use netflix or torrents). The current 2 minute time limit results in a very bad user experience.

Expected Behavior

Since Rancher Desktop is intended for local development, image pulls initiated by K8S should succeed even if they take longer than 2 minutes.

Additional Information

The same issue affects Docker for Desktop (see https://github.com/docker/for-mac/issues/6300)

It can be fixed by changing the kubelet configuration, specifically by increasing the runtimeRequestTimeout setting documented at https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

There are three potential solutions:

1. Find a new global default (like increasing it to 10 minutes or similar, but this might have unintended consequences)
2. Make this number configurable by the user via the UI
3. Give users the ability to modify the kubelet config file on disk and change k3s to use this file when it exists

Rancher Desktop Version

1.3.0

Rancher Desktop K8s Version

v1.20.15

Which container runtime are you using?

moby (docker cli)

What operating system are you using?

macOS

Operating System / Build Version

macOS Monterey 12.3

What CPU architecture are you using?

x64

Linux only: what package format did you use to install Rancher Desktop?

No response

Windows User Only

No response

[jontunjon]

Same problem. The only workaround I have found is to pull the image separately using “docker pull <image>”.

[jgoeres]

We are also running into this - we have a few large images and people in HO. In Minikube, you can set the kubelet parameter “runtime-request-timeout”, which fixes it for us, but some folks would rather use Rancher Desktop. Any chance to set this conveniently?