Unable to build images in Kaniko after update to 0.12.22 (Kaniko + in cluster registry)

Bug

I can no longer build my images in Kaniko after updating to 0.12.22

Current Behavior

Expected behavior

It should build just fine

Reproducible example

Is easily repeatable in my repository but according to @edvald it is not easily reproducible on his side.

To make sure I jumped up and down a version a couple of times and completely removed garden from my cluster and re-installed for both versions.

Works fine on 0.12.21 and fails on 0.12.22

Garden files included in additional context

Workaround

Downgrade back to 0.12.21

Additional context

project.garden.yaml

kind: Project
name: system

environments:
  - name: dev
    defaultNamespace: 'u-${local.env.USER || local.username}'

providers:
  - name: kubernetes
    context: gd1
    #buildMode: cluster-buildkit
    buildMode: kaniko
    namespace:
      name: 'system-u-${local.env.USER || local.username}'
      labels:
        istio-injection: enabled
      annotations:
        field.cattle.io/projectId: "${local.env.GARDEN_RANCHER_PROJECT || '<redacted>'}"
    environments:
      - dev
    storage:
      registry:
        size: 200000
      sync:
        size: 200000


defaultEnvironment: 'dev'

module.garden.yaml

kind: Module
type: exec
name: yarn-build
build:
  command:
    - "./yarn install && ./yarn build"
include:
  - src/**/*.js
  - src/**/*.jsx
  - src/**/*.ts
  - src/**/*.tsx
  - src/**/*.css
  - src/**/*.scss
  - src/**/*.graphql
  - yarn
  - yarn.lock
  - package.json
  - apollo.config.js
  - babel.config.js
  - .eslintrc.json
  - inner_modules
  - jest.config.js
  - postcss.config.js
  - .prettierrc
  - tsconfig.json
  - webpack.config.js
  - webpack.dev.js
  - public/inc
---
kind: Module
type: container
name: system-fpm
dockerfile: Dockerfile
build:
  dependencies:
  - name: yarn-build
    copy:
      - source: public/bundles
        target: public/bundles
hotReload:
  sync:
  - source: src
    target: /var/www/src
  - source: vendor
    target: /var/www/vendor
services:
  - name: system-fpm
    ports:
      - name: fpm
        containerPort: 9000
        servicePort: 9000
---
kind: Module
type: container
name: system
dockerfile: Dockerfile.nginx
build:
  dependencies:
  - name: yarn-build
    copy:
      - source: public/bundles
        target: public/bundles
services:
  - name: system
    ports:
      - name: http
        containerPort: 80
        servicePort: 80
---
kind: Module
type: kubernetes
name: system-gateway
manifests:
  - apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
      name: system-gwy
    spec:
      selector:
        istio: ingressgateway # use istio default controller
      servers:
      - port:
          number: 80
          name: http
          protocol: HTTP
        hosts:
        - ${local.env.USER || local.username}.<redacted>
  - apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: virtual-svc
    spec:
      hosts:
      - ${local.env.USER || local.username}.<redacted>
      gateways:
      - system-gwy
      http:
      - route:
        - destination:
            host: system

Your environment

• OS: Ubuntu 20.04
• How I’m running Kubernetes: GKE
• Garden version: 0.12.22

[edvald]

Aha, I see. I can’t say yet that I fully understand why this would be the case, but that’s good to know. I wouldn’t expect (or like) the registry container to require privileged access.

Maybe it’s rather about the registry chart used being outdated and needing to be set up a bit differently, using a different path for data or something…

checks the version…

Yep, we’ve indeed been using a now-deprecated Helm chart, not tested against recent k8s versions. I don’t know if other versions are out that would address this but that’s certainly a good issue to flag, thanks for hunting this down with us 😃

[edvald]

Btw @carlosjgp I’m pretty sure your issue is resolved now. We’ve got probably three separate issues on this thread 😛 Could you maybe give today’s release a spin?