'rolebindings.authorization.openshift.io "exec" not found' error when start workspace with OS credentials on minishift

Describe the bug

OS OAuth E2E selenium tests had started failing on nightly Jenkins jobs from the next build https://ci.centos.org/view/Devtools/job/devtools-che-nightly-oauth-test/47/console

[ERROR] Failures: [ERROR] > LoginExistedUserWithOpenShiftOAuthTest.checkExistedCheUserOcpProjectCreationAndRemoval:142 » Timeout [ERROR] > LoginNewUserWithOpenShiftOAuthTest.checkNewCheUserOcpProjectCreationAndRemoval:126 » Timeout

Eclipse Che had deployed successfully, but then it was impossible to start workspace with OS credentials:

http://artifacts.ci.centos.org/devtools/che/rc-ocp-oauth-test/30/report/site/failsafe-report.html

There were next commit during the day in Eclipse Che which might had caused regression:

• https://github.com/eclipse/che/commit/db46ad4979f6a82ac2278b410e73659c6463d676

Server logs:

2020-03-26 05:30:08,393[nio-8080-exec-5]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 641]    - Workspace 'user1585200583035/workspaceowv0' with id 'workspacee8wkzi82iwhadgbq' created by user 'user1585200583035'
2020-03-26 05:30:10,982[io-8080-exec-10]  [ERROR] [c.a.c.r.RuntimeExceptionMapper 47]   - Internal Server Error occurred, error time: 2020-03-26 05:30:10
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.30.0.1/apis/authorization.openshift.io/v1/namespaces/user1585200583035-che/rolebindings. Message: rolebindings.authorization.openshift.io "exec" not found. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=authorization.openshift.io, kind=rolebindings, name=exec, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=rolebindings.authorization.openshift.io "exec" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)

Command to deploy Eclipse Che to minishift 3.11:

chectl server:start -a operator -p openshift --k8spodreadytimeout=360000 --chenamespace=eclipse-che --che-operator-cr-yaml=/tmp/custom-resource.yaml

custom-resource.yaml

apiVersion: v1
items:
- apiVersion: org.eclipse.che/v1
  kind: CheCluster
  metadata:
    creationTimestamp: 2020-03-26T05:11:26Z
    finalizers:
    - oauthclients.finalizers.che.eclipse.org
    generation: 17
    name: eclipse-che
    namespace: eclipse-che
    resourceVersion: "3593"
    selfLink: /apis/org.eclipse.che/v1/namespaces/eclipse-che/checlusters/eclipse-che
    uid: 3e28df9e-6f20-11ea-b704-525400a648f9
  spec:
    auth:
      externalIdentityProvider: false
      identityProviderClientId: che-public
      identityProviderImage: quay.io/eclipse/che-keycloak:nightly
      identityProviderPassword: admin
      identityProviderPostgresSecret: che-identity-postgres-secret
      identityProviderRealm: che
      identityProviderSecret: che-identity-secret
      identityProviderURL: http://keycloak-eclipse-che.192.168.42.244.nip.io
      oAuthClientName: eclipse-che-openshift-identity-provider-kbltf2
      oAuthSecret: lxjP9sqONvGZ
      openShiftoAuth: true
      updateAdminPassword: false
    database:
      chePostgresDb: dbche
      chePostgresHostName: postgres
      chePostgresPort: "5432"
      chePostgresSecret: che-postgres-secret
      externalDb: false
    k8s:
      ingressDomain: 192.168.99.101.nip.io
    metrics:
      enable: false
    server:
      allowUserDefinedWorkspaceNamespaces: false
      cheDebug: "false"
      cheFlavor: che
      cheHost: che-eclipse-che.192.168.42.244.nip.io
      cheImageTag: nightly
      cheLogLevel: INFO
      customCheProperties:
        CHE_LIMITS_USER_WORKSPACES_RUN_COUNT: "-1"
      devfileRegistryImage: quay.io/eclipse/che-devfile-registry:nightly
      externalDevfileRegistry: false
      externalPluginRegistry: false
      gitSelfSignedCert: false
      pluginRegistryImage: quay.io/eclipse/che-plugin-registry:nightly
      selfSignedCert: false
      tlsSupport: false
    storage:
      preCreateSubPaths: true
      pvcClaimSize: 1Gi
      pvcStrategy: common
  status:
    cheClusterRunning: Available
    cheURL: http://che-eclipse-che.192.168.42.244.nip.io
    cheVersion: nightly
    dbProvisioned: true
    devfileRegistryURL: http://devfile-registry-eclipse-che.192.168.42.244.nip.io
    keycloakProvisioned: true
    keycloakURL: http://keycloak-eclipse-che.192.168.42.244.nip.io
    openShiftoAuthProvisioned: true
    pluginRegistryURL: http://plugin-registry-eclipse-che.192.168.42.244.nip.io/v3
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
 

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

tests/legacy-e2e/che-selenium-test/selenium-tests.sh \
    --threads=1 \
    --host=${CHE_ROUTE} \
    --port=80 \
    --multiuser \
    --test=org.eclipse.che.selenium.site.ocpoauth.**

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift 3.11
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Installation method

• chectl:next
• che-operator
• minishift-addon
• I don’t know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: ci.centos

Eclipse Che Logs

Additional context

[amisevsk]

@skabashnyuk I assigned severity/P1 to this issue, feel free to revise.

[metlos]

So the fix seems to be to create the roles in namespace instead of cluster-wide and to correctly reference the namespace when referencing the role when creating the role binding. All needs to be done in both the KubernetesWorkspaceServiceAccount and OpenshiftWorkspaceServiceAccount. The latter seems to be already part of https://github.com/eclipse/che/pull/16532 but I would do it as part of a PR for this issue even if we cause a merge conflict by doing that. I will provide the PR soon.