BuildKit tries to reach un-deployed garden-docker-registry

Bug

I am running cluster-buildkit on GKE on cos_containerd against Google Container Registry and I am encountering errors reaching a service which is not available.

Current Behavior

There is a BuildKit deployment created in each namespace.

The BuildKit proxy container tries to reach the garden-docker-registry.garden-system.svc.cluster.local (every 10s) but there is no such service deployed.

2021/04/07 17:59:25 socat[902] E getaddrinfo("garden-docker-registry.garden-system.svc.cluster.local", "NULL", {1,0,1,6}, {}): Name does not resolve

The BuildKit util container is also trying to connect to some service (every 1s) (most likely the same as above):

2021/04/07 17:59:26 [9724] connect from UNDETERMINED (10.51.0.1)

For some reason I saw some garden deployments in the garden-system namespace with an earlier version of the garden-cli. As of right now there is only the namespace and no deployments running inside of it.

Running the following commands is also not re-deploying the services:

garden --env=remote plugins kubernetes uninstall-garden-services
garden --env=remote plugins kubernetes cluster-init

imho there is either something wrong with my garden cluster setup (missing services) or BuildKit should not try to connect to while given an external registry.

Expected behavior

The garden plugins kubernetes cluster-init is redeploying the garden k8s deployments. The BuildKit containers connect to the service and stop logging errors.

Suggested solution(s)

The garden plugins kubernetes cluster-init is redeploying the garden services or trying to fix/ignore the connection issues in the BuildKit deployment.

Sample Configuration

kind: Project
name: some-service
defaultEnvironment: remote
environments:
  - name: remote
providers:
  - name: kubernetes
    environments: ["remote"]
    context: gke_company-cluster_europe-west1-d_cluster
    namespace:
      name: ${local.username}
    buildMode: cluster-buildkit
    deploymentRegistry:
      hostname: "eu.gcr.io"
      namespace: "company-cluster/${local.username}"
    imagePullSecrets:
      - name: gcr-config
        namespace: default

Your environment

• OS: macOS Big Sur
• How I’m running Kubernetes: GKE cluster-buildkit on cos_containerd

garden version 0.12.20

[edvald]

@Dariusch I’ve just pushed a PR (#2375) to get rid of the unnecessary sidecar container. I don’t believe it would have had any adverse effect outside of spamming logs, but it’s still better to mop it up. The cluster-init (and corresponding cleanup) command doesn’t really do anything with BuildKit since it isn’t installed cluster-wide, so that’s quite normal. We’re in fact trying to gradually get rid of the whole cluster-init flow when possible.

@jamesloosli This looks to be an issue with NFS, as @eysi09 suggested. I’d suggest using cluster-buildkit for now, and once #2372 is released you can also use kaniko without worrying about NFS either.

[Dariusch]

Oh and @Dariusch that connect from UNDETERMINED log line is just the Kubernetes health check doing its thing. Shouldn’t be anything to worry about, but if it’s getting in the way we could maybe lower the frequency or the log level in the container.

No its fine, I already excluded those logs from indexing