Eclipse che - volume mount error while launching more than 5 workspace at a time

Describe the bug

Logged in 10 different users at the same time and launched 10 workspaces of each users at a time. 3 - 5 users are able to launch the workspace successfully, for remaining users getting time out error in mount volume, some users keep on loading the workspace and nothing is initialised in log window.

Che version

• latest

Advanced configuration:

Runtime

• kubernetes (include output of kubectl version)
  • kubectl version - Client Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.4”, GitCommit:“e87da0bd6e03ec3fea7933c4b5263d151aafd07c”, GitTreeState:“clean”, BuildDate:“2021-02-18T16:12:00Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“18+”, GitVersion:“v1.18.12-gke.1210”, GitCommit:“199a41188dc0ca5d6d95b1cc7e8ba96e05f9dd0a”, GitTreeState:“clean”, BuildDate:“2021-02-05T18:03:16Z”, GoVersion:“go1.13.15b4”, Compiler:“gc”, Platform:“linux/amd64”}

Installation method

• chectl
  • chectl installation command - sudo chectl server:deploy --installer=helm --platform=k8s --domain=cl-ide.domain.com --multiuser --installer=operator
  • chectl version - chectl/7.27.1 linux-x64 node-v12.21.0

Environment

• my computer
  • Linux
• Cloud
  • GCE
    • Google Kubernete Engine (GKE) - with 2 nodes running
    • Node Configuration - 2 core 8 GB memory Machine with 30 GB hardisk
    • Auto scale is enabaled

Screenshots

Steps to reproduce

• Login 10 users at a time.
• launch 10 different workspace at time.

Expected behavior

It should be able to launch more than 50 workspaces at a time.

[andr-azeez]

Changed the configuration as follows:

  server:
    customCheProperties:
      CHE_INFRA_KUBERNETES_PVC_JOBS_MEMORYLIMIT: 756Mi
      CHE_INFRA_KUBERNETES_WORKSPACE__UNRECOVERABLE__EVENTS: FailedScheduling,MountVolume.SetUpfailed,Failed to pull image,FailedCreate,ReplicaSetCreateError`

Now tested with 20 users and launched 20 workspaces of random stacks. Most of the workspace skipped the error with kubernetes failed to sync configmap cache timed out but able to launch the workspace successfully. Nearly 15 - 17 workspace were launched successfully and nearly 4 - 5 workspace got workspace launch time out error.

Next step is to test about to launch 50 workspace simultaneously, and I will post the result here once the test is done.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.