Push SHA tags for each nightly/next tag (was Start nightly Che workspaces is broken)

Describe the bug

Unable to start new workspace.

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

1. Create new workspace using chectl.
2. Try to start new workspace.
3. See that workspace fails to start with error:

Failed to pull image "quay.io/eclipse/che-theia-endpoint-runtime-binary@sha256:f07e6...ba2": 
rpc error: code = Unknown desc = Error response from daemon: manifest for 
quay.io/eclipse/che-theia-endpoint-runtime-binary@sha256:f0....a2 not found: 
manifest unknown: manifest unknown'

Expected behavior

Workspace should be executed.

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl
  • provide a full command that was used to deploy Eclipse Che (including the output)
  • provide an output of chectl version command
• OperatorHub
• I don’t know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Eclipse Che Logs

unknown, workspaceu5fo1hivysjwxhfe.cpp-dev-7845cb785b-ncnsw
2021-02-01 07:49:17,507[aceSharedPool-0]  [WARN ] [.i.k.KubernetesInternalRuntime 257]  - Failed to start Kubernetes runtime of workspace workspaceu5fo1hivysjwxhfe.
org.eclipse.che.api.workspace.server.spi.InfrastructureException: Unrecoverable event occurred: 'Failed', 'Failed to pull image "quay.io/eclipse/che-theia-endpoint-runtime-binary@sha256:f07e63102eff29069524734d10dab480e34fdace006c284e6e91c5bbb3104ba2": rpc error: code = Unknown desc = Error response from daemon: manifest for quay.io/eclipse/che-theia-endpoint-runtime-binary@sha256:f07e63102eff29069524734d10dab480e34fdace006c284e6e91c5bbb3104ba2 not found: manifest unknown: manifest unknown', 'workspaceu5fo1hivysjwxhfe.cpp-dev-7845cb785b-ncnsw'
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.handleUnrecoverableEvent(KubernetesInternalRuntime.java:1019)
	at org.eclipse.che.workspace.infrastructure.kubernetes.util.UnrecoverablePodEventListener.handle(UnrecoverablePodEventListener.java:44)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.lambda$eventReceived$0(KubernetesDeployments.java:585)
	at java.base/java.util.concurrent.ConcurrentLinkedQueue.forEachFrom(Unknown Source)
	at java.base/java.util.concurrent.ConcurrentLinkedQueue.forEach(Unknown Source)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:585)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:550)
	at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:237)
	at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
2021-02-01 07:49:18,151[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 995]   - Workspace 'admin:cpp-w4qb6' with id 'workspaceu5fo1hivysjwxhfe' start failed
Logs from Feb 1, 

Additional context

[sleshchenko]

So, I should get the new registry after I executed kc rollout restart deployment plugin-registry but I don’t. I face this issue for a long time, is there is no changes in pod configuration, new pod is not run even if new container is available. Seems we need a dedicated issue for that on operator side (maybe it’s actual for helm chart as well). If it’s not expected behavior which would be strange.

To workaround this issue I should

kubectl patch deployment/plugin-registry \
        --patch "{\"spec\":{\"replicas\":0}}
kubectl patch deployment/plugin-registry \
        --patch "{\"spec\":{\"replicas\":1}}

and the issue with broken che-theia-endpoint-runtime-binary disappeared after that.

[nickboldt]

(383M image was UBI based; subsequent builds are alpine only for now.)