Route DNS hostnames not routeable in airgap scenario so che fails to start

Describe the bug

Depending on the network topology or DNS servers, a fully disconnected installation in some instances will not be able to resolve route URLs inside the cluster. This manifests in an issue with the Che server pod trying to retrieve the openid configuration at $PUBLIC_KEYCLOAK_URL/auth/realms/che/.well-known/openid-configuration.

I don’t know exactly how OpenShift does DNS in different environments. I would think that in-cluster traffic would be able to resolve a route properly, but it does not appear to be the case in all scenarios.

curl $KEYCLOAK_ROUTE_URL/auth/realms/che/.well-known/openid-configuration times out, but

curl keycloak.namespace.svc:8080/auth/realms/che/.well-known/openid-configuration succeeds

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

Start a Che installation in a disconnected environment.

Expected behavior

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl
• che-operator 7.4.0
• minishift-addon
• I don’t know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Additional context

[ironcladlou]

By default, on AWS, GCP, and Azure, if cluster DNS zone configuration was provided to the OpenShift installer, OpenShift will manage wildcard DNS records for ingress in the configured zones (assuming ingress is being exposed by a LoadBalancer Service, which is the default on those platforms.)

On other platforms, or if cluster DNS zone configuration is omitted, wildcard DNS records for ingress are not managed and it’s up to the cluster owner to configure DNS to expose ingress (if desired.)

I hope that helps clarify some of the DNS management behavior. I can provide more specific details if someone can help me understand how the problematic clusters are being created (e.g. through the OpenShift installer IPI flow, UPI, etc.)

[davidfestal]

@rhopp @tomgeorge Would it be possible to check with OpenShift teams whether it is expected that typical airgaped OpenShift 4.2 installations would not allow PODs to access external routes ? That seems quite a very hard restriction that would probably make Che fail anyway.