[BUG][Modules] Epiphany's image-registry does not use HTTPS

Describe the bug Communication with Epiphany’s image-registry is insecure.

We use this setting on all Epiphany hosts everywhere where docker runs:

{
    "insecure-registries" : ["mop-module-tests-all-0:5000"],
}

This is the config of docker registry we use on repository hosts (it uses plain HTTP):

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

To Reproduce When AKS tries to pull images from the image-registry, this happens:

  Type     Reason     Age                   From                                      Message
  ----     ------     ----                  ----                                      -------
  Normal   Scheduled  15m                   default-scheduler                         Successfully assigned queue/rabbitmq-cluster-0 to aks-default-31075000-vmss000000
  Normal   Pulling    13m (x4 over 15m)     kubelet, aks-default-31075000-vmss000000  Pulling image "mop-module-tests-all-0:5000/rabbitmq:3.8.3"
  Warning  Failed     13m (x4 over 15m)     kubelet, aks-default-31075000-vmss000000  Failed to pull image "mop-module-tests-all-0:5000/rabbitmq:3.8.3": rpc error: code = Unknown desc = Error response from daemon: Get https://mop-module-tests-all-0:5000/v2/: http: server gave HTTP response to HTTPS client
  Warning  Failed     13m (x4 over 15m)     kubelet, aks-default-31075000-vmss000000  Error: ErrImagePull
  Normal   BackOff    13m (x6 over 15m)     kubelet, aks-default-31075000-vmss000000  Back-off pulling image "mop-module-tests-all-0:5000/rabbitmq:3.8.3"
  Warning  Failed     4m56s (x43 over 15m)  kubelet, aks-default-31075000-vmss000000  Error: ImagePullBackOff

In particular http: server gave HTTP response to HTTPS client is very disturbing 👎

Expected behavior ~Epiphany’s image-registry should at least expose its endpoint on HTTPS.~

• We should use k8s_as_cloud_service variable in application templates to ensure that if we use managed k8s service we use public registry address
• we should add description of that problem to Changelog to “known issues” that it’s impossible to use embedded registry for now and one needs to use external registry (eg.: ACS).

Config files There is no way to change this behavior via config. It’s the only implemented default.

OS (please complete the following information):

• ALL

Cloud Environment (please complete the following information):

• ALL

Additional context N/A

[mkyc]

This variable should be used after #1713 is merged.

[rafzei]

We should deactivate local docker registry usage by applications for now if we use external kubernetes.

Yes, I completely agree. If we wanted to make this registry work with a managed k8s cluster of some kind with enabled autoscaling, that would mean we’d have to modify OS image of the node and add an exception for the private registry to each and every docker daemon runnning there. Epiphany repo would get up AFTER the managed cluster was created, this would mean that such modification would be very complex and inconvenient. -1

Also things like helm roles will not be able to succesfully work with that insecure registry.

Agree. Nothing to-do in this issue, just we wont use local docker registry in aks/eks.