Use a reverse proxy to avoid routes/ingress creation at workspace startup

Description

In the past 2 years of running Che in production we have seen that OpenShift routes do not always fit our needs (we need to bring-up 3 or more routes in a few seconds for every user workspace). The same applies to Kubernetes Ingresses (still in Beta).

Thus the need to investigate alternatives. One proposal is to pre-create one workspace route/ingress for Che server and link it to a reverse proxy that will route all the workspaces traffic (e.g. re-use JWT proxy, envoy or traefik).

That would allow us to:

• limit the number or route/ingress Che needs
• create the route before a user ask to start a workspace

If this approach is validated we could divide the work in 4 steps

• Analyse if the already existent JWT Proxy would satisfy our needs
• All workspaces inbound and outbound traffic should be routed through the JWT Proxy
• Share one JWT Proxy instance amongst all the workspaces of a single user
• Share one JWT Proxy instance amongst all the Che users of a Kube cluster

UPDATE Another important use case is associated to this issue: allow running Che workspaces on OpenShift (where single-host strategy is not available) when using wildcard SSL certificates is not possible.

Implementation

• [Study] Possible implementations for single-host strategy on OpenShift https://github.com/eclipse/che/issues/16702
• Implement a single host version of the happy-path test https://github.com/eclipse/che/issues/16842
• Reuse openshift-router - Impossible
• POC for Traefik - https://github.com/metlos/che-singlehost-poc
• POC for Traefik using CRDs - https://github.com/skabashnyuk/openshift-traefik
• POC for HAProxy - https://github.com/sparkoo/che-singlehost-haproxy-POC
• POC for nginx - #16883
• Investigate a controller able to sync configmaps and send a signal to a gateway process - https://github.com/metlos/cm-bump
• Investigate the best way for hot-reloading the configuration in gateway
  • External controller - #16886
  • In-container controller - #16887
  • Sidecar controller - not working on default Openshift 3.x due to shared process namespace being switched off by default
• Testable POCs
  • Traefik - #16888
  • nginx - should be part of #16883
  • HAProxy - the tests are being implemented using this POC
  • Envoy - #17182
• Do performance tests - #16889
  • Run perf tests for Envoy #17243
• Che Server implementation
  • Enable singlehost strategy to not be ingress-bound #17059
  • Implement gateway-based singlehost strategy for Kubernetes #17060
  • Implement gateway-based singlehost strategy for OpenShift #17061
• ~Implement cookie path rewriting in Traefik ❓ #17062~
• Finalize our config-fetching controller #17063
• Update Helm chart to support gateway-based singlehost mode #17064
• Update the che-operator to support gateway-based singlehost mode #17065
• Document how to deploy gateway singlehost with Chectl #17525
• Keycloak behind the gateway #17809
• Ensure Configubump tool has all necessary CQs open | https://github.com/eclipse/che/issues/17568
• Configubump tool produces incorrect configuration in some cases | https://github.com/eclipse/che/issues/17567

[metlos]

Note that we have concluded our testing of the candidate solutions for a reverse proxy. We chose Traefik and will commence the implementation with #17063 - making our Rust-based POC a fully maintained controller written in Go.

To read more about the selection process and reasoning behind the choice of Traefik, please read through https://www.eclipse.org/lists/che-dev/msg03828.html

[metlos]

In the above mentioned POC repository, we now have 3 POCs implemented:

• haproxy-scripted - this is a vanilla haproxy image operated by oc commands from the test scripts
• nginx-custom-image - this is a custom image using our cm-bump utility and nginxinc/nginx-unprivileged official image of nginx
• traefik-sidecar - this is combination of traefik and cm-bump (we don’t require a custom image in this case because traefik can watch for config changes on its own)

We’re working on haproxy-custom-image which is very similar to nginx-custom-image only with haproxy as the gateway solution. This is to be able to quantify the effect of a custom controller vs externally executed commands.

We’re also working on the testsuite. We’re developing a number of load test scenarios (https://github.com/che-incubator/che-gateway-poc/tree/master/test#testcases). We have not yet started websocket and cookie handling tests which we are going to start once the haproxy-custom-image poc is implemented.