[Solved] Workspace fails to pull redhat image. "no available registry endpoint: could not resolve digest for" Any ideas?

Summary

I recently reinstalled/upgraded eclipse-che multiuser on microk8s using the chectl operator to the newest release 7.28.x, moving from the helm install to the operator install.

The workspaces are not starting up. Inspecting the workspace pod for my user. I get the following error:
The workspace logs on the dashboard don’t convey the problem, the workspace fails to load and times-out.

Failed to pull image “registry.access.redhat.com/ubi8-minimal:8.3-291”: rpc error: code = Unknown desc = failed to resolve image “registry.access.redhat.com/ubi8-minimal:8.3-291”: no available registry endpoint: could not resolve digest for registry.access.redhat.com/ubi8-minimal:8.3-291

Relevant information

` kubectl describe pod/mkdir-workspace1t3hi040f6e7cx5w -n mbd06b-che Name: mkdir-workspace1t3hi040f6e7cx5w Namespace: mbd06b-che Priority: 0 Node: example.com/192.168.86.100 Start Time: Tue, 13 Apr 2021 14:54:53 -0500 Labels: che.workspace_id=workspace1t3hi040f6e7cx5w Annotations: <none> Status: Pending IP: 10.1.44.19 IPs: IP: 10.1.44.19 Containers: mkdir-workspace1t3hi040f6e7cx5w: Container ID: Image: registry.access.redhat.com/ubi8-minimal:8.3-291 Image ID: Port: <none> Host Port: <none> Command: mkdir -m 777 -p /tmp/job_mount/workspace1t3hi040f6e7cx5w/che-logs-che-plugin-broker/ State: Waiting Reason: ErrImagePull Ready: False Restart Count: 0 Limits: memory: 250Mi Requests: memory: 250Mi Environment: <none> Mounts: /tmp/job_mount from claim-che-workspace (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-nl9bs (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: claim-che-workspace: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: claim-che-workspace ReadOnly: false default-token-nl9bs: Type: Secret (a volume populated by a Secret) SecretName: default-token-nl9bs Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

---

Normal Scheduled 3m5s default-scheduler Successfully assigned mbd06b-che/mkdir-workspace1t3hi040f6e7cx5w to example.com Normal Pulling 104s (x4 over 3m5s) kubelet Pulling image “registry.access.redhat.com/ubi8-minimal:8.3-291” Warning Failed 104s (x4 over 3m4s) kubelet Failed to pull image “registry.access.redhat.com/ubi8-minimal:8.3-291”: rpc error: code = Unknown desc = failed to resolve image “registry.access.redhat.com/ubi8-minimal:8.3-291”: no available registry endpoint: could not resolve digest for registry.access.redhat.com/ubi8-minimal:8.3-291 Warning Failed 104s (x4 over 3m4s) kubelet Error: ErrImagePull Warning Failed 76s (x6 over 3m4s) kubelet Error: ImagePullBackOff Normal BackOff 62s (x7 over 3m4s) kubelet Back-off pulling image “registry.access.redhat.com/ubi8-minimal:8.3-291”`

[Mbd06b]

I figured it out… I was using microk8s 1.16/stable channel, which was using an older version of containerd. I migrated to microk8s 1.21/stable which is using Containerd v1.4.4, -> https://microk8s.io/docs/release-notes

An older version of containerd doesn’t seem to work pulling images from that registry, found someone with a similar issue here.
https://qiita.com/yamamoto-febc/items/d2796dabbc59f6650c3d “Solution: OK if you use containerd v1.2.9 / v1.3.0 or later”

using:
‘sudo snap refresh microk8s --channel=1.21/stable’ refresh your kubectl alias… ‘alias kubectl=‘microk8s kubectl’’

[mmorhun]

@Mbd06b thank you for posting detailed answer.