upgrade to 1.14 seems to have broken my access to private registries

I’m under some restrictions about uploading the tarball from microk8s inspect, but

user@orion-a:~$ microk8s.inspect 
Inspecting services
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-etcd is running

args.tar.gz

We’ve been using imagePullSecrets successfully on previous version (1.13.3), but the automatic upgrade to 1.14 both surprised me and seems to have broken the ability to pull from our private registry. (registry is using https://github.com/cesanta/docker_auth and registry:2)

this caused all of our microk8s installs to fail around 17 hours ago when the snap was released.

I checked this issue: https://github.com/containerd/cri/issues/848 but we are using the modern syntax (dockerconfigjson) and haven’t been having an issue prior to yesterday

[ps-feng]

I got this to work!

I checked cri’s source code and saw this:

for _, e := range c.config.Registry.Mirrors[refspec.Hostname()].Endpoints

If we trace that further, the hostname comes from the ‘pull image request’. I wasn’t entirely sure what that is but I figured it could very well be the hostname of the image repository, so then I tried editing /var/snap/microk8s/current/args/containerd-template.toml with:

[plugins.cri.registry.mirrors."192.168.3.25:5000"]
          endpoint = ["http://192.168.3.25:5000"]

Restarted the service, tried deploying bbox.yaml again and it worked!

Bottom line is it seems we can’t just put any arbitrary text after plugins.cri.registry.mirrors., it has to match the host of the registry.

EDIT: after further investigation, I can confirm that it works like stated above. ParseNormalizedName will convert the image name into a fully qualified one that Docker can use unambiguously. When there’s no host name, docker.io is preppended to the name, hence the default [plugins.cri.registry.mirrors."docker.io"]. See test. This makes me wonder about https://github.com/containerd/cri/blob/master/docs/registry.md, which seems misleading as there’s no description about test.secure-registry.io and test.insecure-registry.io and it leads to think that you can put anything there.

[ps-feng]

I too can’t access my private registry and getting https://192.168.3.25:5000/v2/my-busybox/manifests/latest: http: server gave HTTP response to HTTPS client.

My setup is:

• Ubuntu server 18.04 (192.168.3.25)
• docker-ce 18.09.4
• microk8s 1.14.0

And I’ve got a private registry at port 5000. Note that this registry is not in Kubernetes.

I’ve edited /etc/docker/daemon.json to include "insecure-registries" : ["192.168.3.25:5000"], so I’m able to:

docker pull busybox
docker tag busybox 192.168.3.25:5000/my-busybox
docker push 192.168.3.25:5000/my-busybox

My /var/snap/microk8s/current/args/containerd-template.toml also includes

[plugins.cri.registry.mirrors."local.insecure-registry.io"]
          endpoint = ["http://192.168.3.25:5000"]

But then when I try to kubectl apply -f bbox.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: 192.168.3.25:5000/my-busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

I get the dreaded Failed to pull image "192.168.3.25:5000/my-busybox:latest": rpc error: code = Unknown desc = failed to resolve image "192.168.3.25:5000/my-busybox:latest": no available registry endpoint: failed to do request: Head https://192.168.3.25:5000/v2/my-busybox/manifests/latest: http: server gave HTTP response to HTTPS client

After investigating for a while and going through https://github.com/containerd/containerd/issues/2758, I tried doing

$ microk8s.ctr image pull 192.168.3.25:5000/my-busybox:latest
ctr: failed to resolve reference "192.168.3.25:5000/my-busybox:latest": failed to do request: Head https://192.168.3.25:5000/v2/my-busybox/manifests/latest: http: server gave HTTP response to HTTPS client

But then if I try with --plain-http it works!

$ microk8s.ctr image pull --plain-http=true 192.168.3.25:5000/my-busybox:latest
192.168.3.25:5000/my-busybox:latest:                                              resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fc1a6b909f82ce4b72204198d49de3aaf757b3ab2bb823cb6e47c416b97c5985:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:af2f74c517aac1d26793a6ed05ff45b299a037e1a9eefeae5eacda133e70a825:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.3 s                                                                    total:  738.6  (2.4 MiB/s)
unpacking linux/amd64 sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4...
done

So it seems containerd within microk8s is not checking the insecure repository mirror entry. Is this a bug or is there anything else I could do?